RTSP Nonmatching Transport in Server Reply: Debugging Camera SETUP Responses and RTP Mode Mismatch

Some RTSP failures do not return a clean 461 Unsupported Transport. Instead, the client reports "nonmatching transport in server reply", "invalid transport header", "server replied with different transport", or "RTP transport mismatch." This often happens when a camera accepts SETUP but answers with a Transport header that does not match what the client requested or what the client can parse.

Users search for "RTSP nonmatching transport in server reply", "ffmpeg nonmatching transport", "RTSP SETUP transport mismatch", and "camera Transport header invalid" because the stream may work in one player and fail in another. The camera is not simply unreachable. The RTSP transport negotiation is inconsistent.

RtspInspector is useful because the request and response headers must be compared directly.

What a matching SETUP exchange looks like

Client requests TCP interleaved:

Transport: RTP/AVP/TCP;unicast;interleaved=0-1

Server replies with compatible TCP interleaved transport:

Transport: RTP/AVP/TCP;unicast;interleaved=0-1;ssrc=12345678

Client requests UDP:

Transport: RTP/AVP;unicast;client_port=50000-50001

Server replies with UDP ports:

Transport: RTP/AVP;unicast;client_port=50000-50001;server_port=6970-6971

If the server changes transport mode, omits required fields, or returns malformed values, strict clients may fail.

Common mismatch cases

Common examples include:

• Client requests TCP, server replies UDP.
• Client requests UDP, server replies TCP.
• Server omits interleaved=.
• Server returns the wrong channel numbers.
• Server returns client_port values different from the request.
• Server omits server_port for UDP.
• Server returns multicast when client requested unicast.
• Server returns multiple Transport alternatives in an unsupported format.
• Proxy rewrites the request but not the response.

Some clients tolerate these quirks. Others reject them.

Why one player works and another fails

RTSP implementations vary. A tolerant player may accept a malformed or unexpected transport response and continue. A stricter tool may fail because the response violates its expectations.

That does not automatically mean the strict client is wrong. It means the camera or proxy behavior must be inspected.

For professional diagnostics, preserve:

• The requested Transport header.
• The server Transport response.
• Track URL.
• Session ID.
• Whether RTP packets arrive afterward.

Proxy and relay rewriting

RTSP-aware relays may rewrite transport headers to bridge UDP and TCP paths. If the rewrite is incomplete, the downstream client sees a response that does not match its request.

Examples:

• Client requests TCP from relay.
• Relay requests UDP from camera.
• Relay accidentally forwards camera's UDP Transport response downstream.

The client reports nonmatching transport even though the camera and relay each did something partially valid.

Debug checklist

Use this process:

1. Capture SETUP request.
2. Capture SETUP response.
3. Compare transport protocol: UDP, TCP interleaved, multicast.
4. Compare unicast/multicast.
5. Compare client ports, server ports, and interleaved channels.
6. Check whether a proxy/restreamer is in the path.
7. Check whether later RTP packets follow the response mapping.
8. Compare a tolerant player and strict client using packet evidence.
9. Test direct camera URL if possible.
10. Report the exact Transport pair to the vendor.

Final diagnosis

"Nonmatching transport in server reply" means SETUP negotiation produced an incompatible or malformed Transport response. The camera or relay may be changing RTP mode, omitting fields, or returning values the client cannot safely use.

RtspInspector helps by making the transport request/response pair visible, which is the only reliable way to diagnose this class of RTSP failure.