H.264 FU-A RTP Fragmentation Fix: Why Your RTSP Video Is Broken (Packet Loss, NAL Reassembly)

H.264 over RTP breaks in ways that look like decoder bugs but are really packetization problems. The RTSP session works. DESCRIBE returns valid SDP. SETUP succeeds. PLAY returns 200 OK. RTP packets arrive with the correct payload type mapped to H.264. Still, the video shows block corruption, freezes, black frames, or decoder errors.

The errors look like this in your logs:

[h264 @ 0x...] invalid NAL unit size
[h264 @ 0x...] missing picture in access unit
[h264 @ 0x...] non-existing PPS 0 referenced
[h264 @ 0x...] decode_slice_header error
[h264 @ 0x...] no frame!

When RTP packets arrive but the decoder can't produce clean video, the problem is almost always at the fragmentation layer — the layer between the network and the decoder that reassembles RTP packets into complete NAL units.

The fast answer

If you see H.264 decoder errors but RTP packets are arriving:

1. Check RTP sequence numbers for gaps (packet loss)
2. Check FU-A start and end bits (fragmentation boundaries)
3. Check the RTP marker bit (access unit boundaries)
4. Check that all fragments of each NAL unit arrive
5. Compare UDP vs TCP transport to isolate network loss

If any FU-A fragment is missing, the decoder receives a broken NAL unit. The fix is either reducing packet loss (network layer) or reducing fragment count (encoder layer).

Understanding FU-A: how H.264 crosses the network

Why fragmentation exists

An H.264 NAL unit can be larger than the network MTU (typically 1500 bytes). A 1080p IDR frame can be 50-100 KB — far too large for a single packet. The RTP payload format for H.264 (RFC 6184) defines fragmentation units (FU-A) to split large NAL units across multiple RTP packets.

The structure of an FU-A packet

Each FU-A RTP packet contains:

Byte 0: FU indicator
  bit 7: F (forbidden_zero_bit) — normally 0
  bits 6-5: NRI (nal_ref_idc) — priority
  bits 4-0: Type = 28 (FU-A)
Byte 1: FU header
  bit 7: S (Start) — 1 for first fragment
  bit 6: E (End) — 1 for last fragment
  bit 5: R (Reserved) — always 0
  bits 4-0: Type — original NAL unit type (1=non-IDR, 5=IDR, 7=SPS, 8=PPS)
Byte 2+: Fragment payload — the actual NAL unit data

A complete FU-A sequence for one NAL unit:

Packet 1: FU indicator (type=28) | FU header (S=1, E=0, type=5) | payload[0..N]
Packet 2: FU indicator (type=28) | FU header (S=0, E=0, type=5) | payload[N+1..M]
Packet 3: FU indicator (type=28) | FU header (S=0, E=0, type=5) | payload[M+1..P]
Packet 4: FU indicator (type=28) | FU header (S=0, E=1, type=5) | payload[P+1..END]

All fragments share:

• Same RTP timestamp (same video frame)
• Same FU indicator NRI value
• Same FU header type (original NAL type)
• Consecutive RTP sequence numbers (in order)

The RTP marker bit

The RTP marker bit signals the final RTP packet of an access unit, not merely the end of every fragmented NAL unit. A FU-A packet with E=1 carries the marker only when that NAL unit is also the last NAL unit in the access unit. A marker on an earlier FU fragment creates a false access-unit boundary; E=1 with the marker clear can be valid when another NAL unit follows.

Failure modes: what breaks and why

Failure 1: Missing middle fragment

Expected:  1000(S)  1001(M)  1002(M)  1003(E|marker)
Received:  1000(S)  1001(M)  [LOST]   1003(E|marker)

The reassembler sees a start fragment, one middle fragment, then an end fragment — but the payload bytes don't connect. The reassembled NAL unit has a gap.

Symptoms:

• Block corruption in a horizontal band of the frame
• Decoder reports "invalid NAL unit size"
• Corruption limited to one access unit (clears at next IDR)

Failure 2: Missing start fragment

Expected:  1000(S)  1001(M)  1002(E)
Received:  [LOST]   1001(M)  1002(E)

Without the start fragment, the reassembler doesn't know the original NAL unit type. It can't determine if this is an IDR slice, a non-IDR slice, or parameter set data.

Symptoms:

• Decoder drops the entire access unit
• Video freezes until next IDR
• May see "non-existing PPS" or "no frame" errors

Failure 3: Missing end fragment

Expected:  1000(S)  1001(M)  1002(E|marker)
Received:  1000(S)  1001(M)  [LOST]

The reassembler never sees the end fragment. The NAL unit boundary is never confirmed. The decoder waits for more data that never arrives.

Symptoms:

• Video freezes
• Decoder reports "missing picture in access unit"
• Playback stalls until next complete access unit arrives

Failure 4: Out-of-order fragments

Expected:  1000(S)  1001(M)  1002(E)
Received:  1000(S)  1002(E)  1001(M)

Some reassemblers handle reordering. Many don't. If the end fragment arrives before the middle fragment, a naive reassembler may close the NAL unit early.

Symptoms:

• Frame corruption similar to missing middle fragment
• Intermittent (depends on network jitter)

Failure 5: MTU-induced fragmentation at the network layer

Ethernet MTU is 1500 bytes. IP + UDP + RTP headers consume ~40 bytes, leaving ~1460 bytes for payload. If a FU-A fragment exceeds this, IP fragmentation occurs — splitting the RTP packet at the IP layer into multiple IP fragments.

If any IP fragment is lost, the entire RTP packet is lost. This compounds the FU-A fragmentation problem: you have fragmentation at two layers, and loss at either layer breaks the NAL unit.

Check: If you see IP fragments in your capture alongside FU-A fragments, your encoder is producing FU-A fragments larger than the path MTU. Lower the encoder's MTU setting or reduce the NAL unit size.

Diagnostic workflow

Step 1: Confirm H.264 payload mapping

From the RTSP DESCRIBE response, find the video track:

m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=4D0029;sprop-parameter-sets=...

• packetization-mode=1 means non-interleaved mode (FU-A is supported)
• packetization-mode=0 means single NAL unit mode (no fragmentation — very limited)

Step 2: Track RTP sequence numbers

In your capture, look at RTP sequence numbers for the video track. Gaps indicate packet loss:

Seq 1000: FU-A Start, NAL type 5 (IDR)
Seq 1001: FU-A middle
Seq 1003: FU-A End, marker=1   ← gap at 1002

The gap at 1002 is a missing fragment. If this is an IDR frame, the corruption will affect every frame until the next IDR.

Step 3: Inspect FU-A headers

For each RTP packet on the video track, decode bytes 0 and 1 of the payload:

Byte 0 (FU indicator):
  0x7C = NRI=3, Type=28 (FU-A)
  
Byte 1 (FU header):
  0x85 = S=1, E=0, R=0, Type=5 (IDR slice)  ← start fragment of IDR
  0x45 = S=0, E=0, R=0, Type=5               ← middle fragment
  0x65 = S=0, E=1, R=0, Type=5               ← end fragment; marker only if this NAL ends the access unit

Wrong values to look for:

• S=1 on a non-start packet → reassembler confusion
• Marker set before E=1 → false access-unit boundary
• E=1 with the marker clear may be valid when another NAL unit follows
• Type changes mid-sequence → encoder bug or stream corruption

Step 4: Verify complete reassembly

For each NAL unit:

1. Find the start fragment (S=1)
2. Follow consecutive sequence numbers until the end fragment (E=1)
3. Count fragments; verify no sequence gaps
4. Compare the marker with the access-unit boundary; do not require it on every FU-A end fragment
5. Verify all fragments have the same timestamp and NAL type

Step 5: Compare transport modes

Run the same stream over UDP and TCP interleaved. If video is clean over TCP but corrupted over UDP, the problem is network packet loss, not encoder or decoder issues.

Step 6: IDR vs non-IDR analysis

IDR frames are larger and require more FU-A fragments per NAL unit. More fragments = more exposure to packet loss. If corruption appears mostly on scene changes or every few seconds (at the IDR interval), the IDR frames are the victim.

Fix options:

• Lower the IDR frame size (reduce resolution or bitrate for keyframes)
• Increase the IDR interval (fewer large frames, but slower recovery from loss)
• Reduce MTU for FU-A at the encoder (more fragments, but each is smaller and less likely to trigger IP fragmentation)

Recovery strategies

At the reassembler

• Drop damaged access units entirely rather than feeding partial data to the decoder
• Request a new IDR frame via RTCP (if supported by the camera)
• Wait for the next IDR frame (decoder will recover automatically)

At the encoder

• Lower the NAL unit size limit to reduce fragments per access unit
• Use periodic intra-refresh instead of full IDR frames (smaller keyframes)
• Enable FEC (forward error correction) if the RTP stack supports it

At the network

• Switch from UDP to TCP interleaved if packet loss is unavoidable
• Ensure path MTU discovery works (avoid IP fragmentation)
• Prioritize RTP traffic on congested links

When it's not FU-A

Not all H.264 corruption is FU-A fragmentation. Check:

• SPS/PPS missing: Decoder errors like "non-existing PPS" — check SDP sprop-parameter-sets
• Wrong profile/level: Decoder can't handle the stream complexity
• Encoder bug: The encoder produces invalid NAL unit syntax
• Bitstream corruption at source: The camera's encoder is broken, not the network

If the RTP packets arrive with perfect sequence numbers and correct FU-A headers, but the decoder still fails, the problem is in the H.264 bitstream itself, not the transport.