RTSP Nonmatching Transport in Server Reply: Debugging Camera SETUP Responses and RTP Mode Mismatch

Some RTSP failures do not return a clean 461 Unsupported Transport. Instead, the client reports "nonmatching transport in server reply", "invalid transport header", "server replied with different transport", or "RTP transport mismatch." This often happens when a camera accepts SETUP but answers with a Transport header that does not match what the client requested or what the client can parse.

When ffmpeg reports “nonmatching transport in server reply” but another player works, compare the client's SETUP Transport alternatives with the server's selected profile, unicast/multicast mode, client/server ports, and interleaved channels. The RTSP transcript can prove that a camera returned a different or malformed Transport header rather than a reachability failure. Whether a tolerant player safely compensates for the violation is an implementation question beyond the protocol exchange.

RTSP Inspector is useful because the request and response headers must be compared directly.

What a matching SETUP exchange looks like

Client requests TCP interleaved:

Transport: RTP/AVP/TCP;unicast;interleaved=0-1

Server replies with compatible TCP interleaved transport:

Transport: RTP/AVP/TCP;unicast;interleaved=0-1;ssrc=12345678

Client requests UDP:

Transport: RTP/AVP;unicast;client_port=50000-50001

Server replies with UDP ports:

Transport: RTP/AVP;unicast;client_port=50000-50001;server_port=6970-6971

If the server changes transport mode, omits required fields, or returns malformed values, strict clients may fail.

Common mismatch cases

Common examples include:

• Client requests TCP, server replies UDP.
• Client requests UDP, server replies TCP.
• Server omits interleaved=.
• Server returns the wrong channel numbers.
• Server returns client_port values different from the request.
• Server omits server_port for UDP.
• Server returns multicast when client requested unicast.
• Server returns multiple Transport alternatives in an unsupported format.
• Proxy rewrites the request but not the response.

Some clients tolerate these quirks. Others reject them.

Why one player works and another fails

RTSP implementations vary. A tolerant player may accept a malformed or unexpected transport response and continue. A stricter tool may fail because the response violates its expectations.

That does not automatically mean the strict client is wrong. It means the camera or proxy behavior must be inspected.

For professional diagnostics, preserve:

• The requested Transport header.
• The server Transport response.
• Track URL.
• Session ID.
• Whether RTP packets arrive afterward.

Proxy and relay rewriting

RTSP-aware relays may rewrite transport headers to bridge UDP and TCP paths. If the rewrite is incomplete, the downstream client sees a response that does not match its request.

Examples:

• Client requests TCP from relay.
• Relay requests UDP from camera.
• Relay accidentally forwards camera's UDP Transport response downstream.

The client reports nonmatching transport even though the camera and relay each did something partially valid.

Debug checklist

Use this process:

1. Capture SETUP request.
2. Capture SETUP response.
3. Compare transport protocol: UDP, TCP interleaved, multicast.
4. Compare unicast/multicast.
5. Compare client ports, server ports, and interleaved channels.
6. Check whether a proxy/restreamer is in the path.
7. Check whether later RTP packets follow the response mapping.
8. Compare a tolerant player and strict client using packet evidence.
9. Test direct camera URL if possible.
10. Report the exact Transport pair to the vendor.

Final diagnosis

"Nonmatching transport in server reply" means SETUP negotiation produced an incompatible or malformed Transport response. The camera or relay may be changing RTP mode, omitting fields, or returning values the client cannot safely use.

RTSP Inspector helps by making the transport request/response pair visible, which is the only reliable way to diagnose this class of RTSP failure.