RTSP 461 Unsupported Transport: Fixing SETUP Failed, UDP vs TCP, and Camera Transport Modes

RTSP/1.0 461 Unsupported Transport is one of the most searchable RTSP errors because it appears in ffmpeg, Frigate, Scrypted, HomeKit bridges, NVR integrations, Android clients, and custom camera tools. The log often says method SETUP failed: 461 Unsupported Transport. The camera may accept OPTIONS and DESCRIBE, but when the client tries to set up RTP media transport, the camera rejects the requested transport mode.

Users search for "RTSP 461 Unsupported Transport", "ffmpeg method SETUP failed 461", "RTSP UDP vs TCP camera", "RTP AVP TCP interleaved unsupported", and "camera SETUP failed unsupported transport" because the problem is not the RTSP URL alone. It is the transport negotiation between client and server.

RtspInspector is useful here because the decisive evidence is the SETUP request and the Transport header.

What SETUP does

After DESCRIBE, the client knows the stream tracks from SDP. Then it sends SETUP for each track to negotiate media transport.

UDP example:

SETUP rtsp://camera/stream/trackID=1 RTSP/1.0
Transport: RTP/AVP;unicast;client_port=50000-50001

TCP interleaved example:

SETUP rtsp://camera/stream/trackID=1 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=0-1

If the camera does not support the requested mode, it may return:

RTSP/1.0 461 Unsupported Transport

UDP-only and TCP-only behavior

Many cameras support both UDP RTP and TCP interleaved RTP. Some do not. Some older cameras support only UDP. Some cloud relays or restreamers support only TCP interleaved. Some NVR paths behave differently from direct camera paths.

Symptoms:

• UDP SETUP fails but TCP works.
• TCP interleaved SETUP fails but UDP works.
• Direct camera works, restream URL fails.
• Main stream supports one mode, sub-stream supports another.
• Client retries from UDP to TCP and still fails because the server rejects both requested formats.

The fix is not "always use TCP" or "always use UDP." The fix is to identify what the server actually accepts.

Transport header syntax matters

Some RTSP servers are strict. They may reject headers that are valid in theory but not accepted by firmware.

Compare:

Transport: RTP/AVP;unicast;client_port=50000-50001

and:

Transport: RTP/AVP/UDP;unicast;client_port=50000-50001

Some devices treat these differently. Some reject multicast. Some reject a client port range outside expected bounds. Some reject interleaved channel numbers not starting from zero.

RtspInspector should preserve the exact header string, not just a simplified "TCP" or "UDP" label.

Restreamers and proxy paths

When RTSP comes through a restreamer, NVR, or media bridge, the upstream and downstream transport capabilities may not match. The camera may support UDP, but the restreamer only exposes TCP. Or the restreamer may accept a SETUP request but fail when mapping media channels.

If a URL points to 127.0.0.1:8554 or another restreamer instead of the camera, diagnose that server's RTSP behavior, not only the camera's.

Debug checklist

Use this workflow:

1. Capture DESCRIBE and SDP.
2. Identify the track URL used in SETUP.
3. Inspect the exact Transport header.
4. Record the 461 Unsupported Transport response.
5. Try UDP and TCP interleaved modes deliberately.
6. Check whether the camera supports multicast, UDP unicast, or TCP interleaved.
7. Compare main stream and sub-stream.
8. Compare direct camera URL and NVR/restream URL.
9. Avoid codec debugging until SETUP succeeds.
10. Preserve the exact request/response pair for vendor support.

Final diagnosis

RTSP 461 Unsupported Transport means the server rejected the media transport requested during SETUP. The root cause is usually UDP/TCP mode mismatch, strict Transport header syntax, unsupported interleaved mode, unsupported UDP mode, restreamer behavior, or track-specific server limitations.

RtspInspector helps diagnose this at the correct layer: RTSP transport negotiation before RTP packets ever appear.