2026-06-02

RTSP Nonmatching Transport: SETUP Reply Fix

How to debug RTSP nonmatching transport errors when the camera replies with a different RTP transport mode, wrong interleaved channels, missing ports, or incompatible SETUP response.

Some RTSP failures do not return a clean 461 Unsupported Transport. Instead, the client reports "nonmatching transport in server reply", "invalid transport header", "server replied with different transport", or "RTP transport mismatch." This often happens when a camera accepts SETUP but answers with a Transport header that does not match what the client requested or what the client can parse.

When ffmpeg reports “nonmatching transport in server reply” but another player works, compare the client's SETUP Transport alternatives with the server's selected profile, unicast/multicast mode, client/server ports, and interleaved channels. The RTSP transcript can prove that a camera returned a different or malformed Transport header rather than a reachability failure. Whether a tolerant player safely compensates for the violation is an implementation question beyond the protocol exchange.

RTSP Inspector is useful because the request and response headers must be compared directly.

What a matching SETUP exchange looks like

Client requests TCP interleaved:

Transport: RTP/AVP/TCP;unicast;interleaved=0-1

Server replies with compatible TCP interleaved transport:

Transport: RTP/AVP/TCP;unicast;interleaved=0-1;ssrc=12345678

Client requests UDP:

Transport: RTP/AVP;unicast;client_port=50000-50001

Server replies with UDP ports:

Transport: RTP/AVP;unicast;client_port=50000-50001;server_port=6970-6971

If the server changes transport mode, omits required fields, or returns malformed values, strict clients may fail.

Common mismatch cases

Common examples include:

Client requests TCP, server replies UDP.
Client requests UDP, server replies TCP.
Server omits interleaved=.
Server returns the wrong channel numbers.
Server returns client_port values different from the request.
Server omits server_port for UDP.
Server returns multicast when client requested unicast.
Server returns multiple Transport alternatives in an unsupported format.
Proxy rewrites the request but not the response.

Some clients tolerate these quirks. Others reject them.

Why one player works and another fails

RTSP implementations vary. A tolerant player may accept a malformed or unexpected transport response and continue. A stricter tool may fail because the response violates its expectations.

That does not automatically mean the strict client is wrong. It means the camera or proxy behavior must be inspected.

For professional diagnostics, preserve:

The requested Transport header.
The server Transport response.
Track URL.
Session ID.
Whether RTP packets arrive afterward.

Proxy and relay rewriting

RTSP-aware relays may rewrite transport headers to bridge UDP and TCP paths. If the rewrite is incomplete, the downstream client sees a response that does not match its request.

Examples:

Client requests TCP from relay.
Relay requests UDP from camera.
Relay accidentally forwards camera's UDP Transport response downstream.

The client reports nonmatching transport even though the camera and relay each did something partially valid.

Debug checklist

Use this process:

Capture SETUP request.
Capture SETUP response.
Compare transport protocol: UDP, TCP interleaved, multicast.
Compare unicast/multicast.
Compare client ports, server ports, and interleaved channels.
Check whether a proxy/restreamer is in the path.
Check whether later RTP packets follow the response mapping.
Compare a tolerant player and strict client using packet evidence.
Test direct camera URL if possible.
Report the exact Transport pair to the vendor.

Final diagnosis

"Nonmatching transport in server reply" means SETUP negotiation produced an incompatible or malformed Transport response. The camera or relay may be changing RTP mode, omitting fields, or returning values the client cannot safely use.

RTSP Inspector helps by making the transport request/response pair visible, which is the only reliable way to diagnose this class of RTSP failure.

Normalize the SETUP pair before declaring a mismatch

The word “nonmatching” is useful only when both sides of the exchange are visible. Put the request Transport header and response Transport header next to one another, retaining their original order and parameters. Then compare the selected transport family, unicast or multicast mode, interleaved channels, client ports, server ports, mode, and SSRC. Do not reduce the exchange to “TCP” or “UDP”; the fields determine whether a receiver can safely associate later media with the session.

Client requests	Server replies	Why a strict client may reject it	Evidence-led next action
TCP interleaved `0-1`	UDP server ports	Transport family changed	Check relay translation or offer only supported alternative
UDP client ports	TCP interleaved channels	Receiver has no agreed UDP mapping	Confirm whether a proxy terminated UDP upstream
TCP interleaved `0-1`	TCP interleaved `2-3`	Channel map differs from request	Inspect whether client can accept selected channels safely
UDP unicast	Multicast	Delivery model changed	Check explicit multicast policy and downstream reachability
UDP client ports `50000-50001`	Different client ports	Response may be malformed or rewritten	Compare relay behavior and actual socket mapping
Valid-looking SETUP but no RTP	Matching mapping, then media evidence	Negotiation may be accepted but delivery still fails	Inspect RTP/RTCP after PLAY

This is not a request for clients to accept malformed headers blindly. A tolerant player may continue by guessing, while a recorder or forensic tool must decide whether it can map RTP safely. The protocol transcript tells a support team exactly which decision was made.

TCP interleaved channels are a mapping contract

For RTP over TCP, the interleaved=a-b pair identifies the channels that carry RTP and RTCP within the RTSP connection. A request for 0-1 and a reply of 2-3 is not automatically unusable, but it is a selected mapping that the client must consciously accept. If media appears later on channels different from the response, the issue may be a server, relay, or parser defect rather than ordinary packet loss.

Capture the SETUP response, the following interleaved frame channel bytes, and the next RTCP packet if present. The RTSP TCP interleaved channel mismatch guide helps separate an incorrect channel map from a stream that simply has no video because PLAY or codec setup failed.

Do not compare a TCP interleaved exchange with a separate UDP attempt as if both must produce identical numbers. The relevant comparison is whether the response selected a compatible mode for that request and whether later media follows that selected mapping.

UDP port fields need the same discipline

For UDP, the client normally proposes a receive port pair and the server returns the ports it will use. A response with missing server ports, rewritten client ports, or an unexpected multicast destination may be malformed, but first establish whether an NVR, proxy, or NAT helper sits between the two endpoints. A relay can legitimately use UDP upstream and TCP interleaved downstream; the defect occurs when it exposes the upstream response to the downstream client without translating it.

Record the client-visible address, the Transport pair, any relay identity, and whether RTP appears at the ports the response describes. Do not publish private addresses or port mappings in an external ticket. A direct-camera test can be valuable, but it only proves a path difference when the same profile and transport preference are used.

Offer alternatives intentionally

Some clients send a comma-separated set of Transport alternatives. A server should select one compatible alternative, but products disagree on parsing and formatting. Preserve the full request rather than only the first token. If the camera accepts TCP interleaved but rejects UDP, test a single explicit TCP request and then a single explicit UDP request. This removes ambiguity about which alternative triggered the response.

When a server returns 461, the issue is an unsupported proposal rather than a nonmatching success response; use the RTSP 461 transport workflow. When it returns 200 with incompatible fields, retain the complete pair and inspect the next media frames. Those are different support cases.

Reproduce without accidentally changing state

Use one track and one clean session per transport test. Start with OPTIONS and DESCRIBE, then SETUP using exactly one mode. Record the returned Session and Transport headers before PLAY. Close the session before switching from TCP to UDP or back again. This avoids carrying an old Session ID or a prior track mapping into the next result.

If a mismatch appears only after a reconnect, include the connection boundary in the case. A stale session can produce confusing errors that look like transport behavior. If SETUP succeeds but later PLAY fails with 454, inspect session state separately before deciding that the returned Transport was the root cause.

Nonmatching transport FAQ

Why does ffmpeg reject a response that VLC accepts?

Clients choose different levels of tolerance for unexpected parameters, channel mappings, and malformed responses. A permissive player may guess enough to render video; a stricter client may refuse to avoid receiving or recording media on an unsafe mapping. The request/response pair identifies the compatibility issue without declaring either client universally wrong.

Can I force TCP to avoid this error?

Only if the camera or relay supports TCP interleaving and returns a compatible mapping. Forcing TCP cannot repair a relay that forwards an upstream UDP response unchanged, a wrong channel pair, or a malformed header. Capture the result of the explicit request.

Does a matching Transport header prove video will arrive?

No. It proves only that SETUP negotiated a mapping. PLAY, RTP delivery, codec configuration, and downstream decode remain separate checkpoints. Inspect RTP and RTCP after PLAY before calling the stream healthy.

What should a camera vendor receive?

Provide redacted SETUP request and response headers, the selected stream profile, whether a proxy is present, the Session header, and the first RTP or interleaved frame observation. That is enough to reproduce a transport-format issue without exposing credentials.

Can a malformed reply still deliver media?

It can, especially when a permissive client guesses the intended port pair or interleaved channels. That behavior is not proof that every client can make the same safe guess. Retain the actual media observation alongside the malformed header, then decide whether the deployment can tolerate the vendor quirk or needs a relay, firmware, or client configuration change that restores an explicit mapping.

Is a direct-camera test enough to close the issue?

It is strong evidence about the path, not an automatic answer. Repeat the same profile, same transport preference, and same client behavior through the relay and directly at the camera. If only the relay path changes the Transport reply, the case has a precise boundary. If both replies differ from the request, preserve both pairs for the camera vendor.

Related guides

Continue with the same-language pages below. They cover adjacent stages without changing the canonical owner of this topic: