RTSP 461 Unsupported Transport Fix: SETUP Failed, UDP vs TCP, ffmpeg Camera Transport Error

An ffmpeg or NVR session may complete OPTIONS and DESCRIBE, then stop at method SETUP failed: 461 Unsupported Transport. That sequence says the RTSP URL and authentication reached the server; the disputed part is whether the camera accepts the requested UDP ports, multicast mode, or RTP/AVP/TCP interleaved transport. Comparing the exact Transport request with the 461 response can prove a negotiation mismatch, while it cannot show undocumented transport modes that the firmware never advertises.

RTSP Inspector is useful here because the decisive evidence is the SETUP request and the Transport header.

What SETUP does

After DESCRIBE, the client knows the stream tracks from SDP. Then it sends SETUP for each track to negotiate media transport.

UDP example:

SETUP rtsp://camera/stream/trackID=1 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=50000-50001

TCP interleaved example:

SETUP rtsp://camera/stream/trackID=1 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=0-1

If the camera does not support the requested mode, it may return:

RTSP/1.0 461 Unsupported Transport

UDP-only and TCP-only behavior

Many cameras support both UDP RTP and TCP interleaved RTP. Some do not. Some older cameras support only UDP. Some cloud relays or restreamers support only TCP interleaved. Some NVR paths behave differently from direct camera paths.

Symptoms:

• UDP SETUP fails but TCP works.
• TCP interleaved SETUP fails but UDP works.
• Direct camera works, restream URL fails.
• Main stream supports one mode, sub-stream supports another.
• Client retries from UDP to TCP and still fails because the server rejects both requested formats.

The fix is not "always use TCP" or "always use UDP." The fix is to identify what the server actually accepts.

Transport header syntax matters

Some RTSP servers are strict. They may reject headers that are valid in theory but not accepted by firmware.

Compare:

Transport: RTP/AVP;unicast;client_port=50000-50001

and:

Transport: RTP/AVP/UDP;unicast;client_port=50000-50001

Some devices treat these differently. Some reject multicast. Some reject a client port range outside expected bounds. Some reject interleaved channel numbers not starting from zero.

RTSP Inspector should preserve the exact header string, not just a simplified "TCP" or "UDP" label.

Restreamers and proxy paths

When RTSP comes through a restreamer, NVR, or media bridge, the upstream and downstream transport capabilities may not match. The camera may support UDP, but the restreamer only exposes TCP. Or the restreamer may accept a SETUP request but fail when mapping media channels.

If a URL points to 127.0.0.1:8554 or another restreamer instead of the camera, diagnose that server's RTSP behavior, not only the camera's.

Debug checklist

Use this workflow:

1. Capture DESCRIBE and SDP.
2. Identify the track URL used in SETUP.
3. Inspect the exact Transport header.
4. Record the 461 Unsupported Transport response.
5. Try UDP and TCP interleaved modes deliberately.
6. Check whether the camera supports multicast, UDP unicast, or TCP interleaved.
7. Compare main stream and sub-stream.
8. Compare direct camera URL and NVR/restream URL.
9. Avoid codec debugging until SETUP succeeds.
10. Preserve the exact request/response pair for vendor support.

Final diagnosis

RTSP 461 Unsupported Transport means the server rejected the media transport requested during SETUP. The root cause is usually UDP/TCP mode mismatch, strict Transport header syntax, unsupported interleaved mode, unsupported UDP mode, restreamer behavior, or track-specific server limitations.

RTSP Inspector helps diagnose this at the correct layer: RTSP transport negotiation before RTP packets ever appear.