2026-06-02

MTU Black Hole and Fragmentation PCAP Analysis: Finding PMTUD Failures, MSS Problems, and Stalled TCP

How to diagnose MTU black holes, path MTU discovery failure, TCP MSS mismatch, fragmentation, ICMP blocked messages, VPN tunnels, and stalled connections in packet captures.

When ping, DNS, and a TCP handshake succeed but a large TLS response, upload, VPN flow, or high-bitrate stream stalls, the path may have an MTU black hole rather than a general outage. Look for a repeatable payload-size boundary, oversized retransmissions, IPv4 fragmentation, IPv6 Packet Too Big, or missing ICMP feedback to separate Path MTU Discovery failure from an ordinary TCP MSS problem. Packet sizes in a capture are evidence, but offload and tunnel overhead can make the local trace differ from what actually crossed the wire.

A packet capture can show whether large packets are sent, fragmented, dropped, retransmitted, or blocked by missing ICMP feedback. But the trace must be interpreted carefully because capture point, offload, and tunnel overhead can distort the visible packet sizes.

PCAP Surgery is useful here because MTU investigations often need focused traces: one TCP conversation, original timing, ICMP messages, retransmissions, MSS negotiation, and enough packets before and after the stall.

What MTU means

MTU is the maximum transmission unit on a link. Ethernet commonly uses 1500 bytes. VPNs, PPPoE, tunnels, overlays, and cloud networking can reduce the effective path MTU because they add headers.

If a sender transmits packets larger than a path can carry, one of two things must happen:

Packets are fragmented where allowed.
Packets are dropped and the sender must learn a smaller path MTU.

Path MTU Discovery relies on ICMP messages to report that a packet was too large. If those ICMP messages are blocked, the sender may keep sending packets that disappear. That is the classic MTU black hole.

Symptoms of MTU black hole

Common symptoms include:

TCP connection opens but large transfer stalls.
HTTPS handshake stalls after ClientHello or ServerHello.
Login page loads but file upload fails.
VPN connects but some sites do not load.
SSH works until large output appears.
API calls with small payloads work, large payloads fail.
RTSP control works but media or large interleaved packets behave poorly.
Retransmissions repeat with similar packet sizes.

The key pattern is size sensitivity. Small packets pass. Larger packets fail.

MSS negotiation

TCP Maximum Segment Size is negotiated during the SYN exchange:

Client -> Server: SYN, MSS 1460
Server -> Client: SYN-ACK, MSS 1460

MSS tells the peer the largest TCP payload segment it should send. For normal Ethernet with 1500 MTU, MSS is often 1460 for IPv4 TCP without options. Through a VPN, a lower MSS may be needed.

If MSS is too high for the real path, large segments may be dropped later.

In a capture, inspect the SYN and SYN-ACK. If the path includes a VPN or tunnel but MSS remains 1460, MSS clamping may be missing.

ICMP fragmentation needed

For IPv4 with Don't Fragment set, a router that cannot forward a packet should send an ICMP "fragmentation needed" message. For IPv6, packet-too-big messages are essential because routers do not fragment packets in the same way.

If the capture shows:

Large packet sent.
ICMP packet-too-big received.
Sender reduces packet size.
Transfer continues.

then Path MTU Discovery is working.

If the capture shows:

Large packet sent repeatedly.
No ICMP feedback.
Retransmissions repeat.
Transfer stalls.

then an MTU black hole is plausible.

Fragmentation evidence

Fragmentation itself is not always a failure. But fragmented traffic can be dropped by firewalls, NAT devices, or overloaded paths. Large UDP payloads are especially vulnerable because there is no TCP retransmission at the transport layer.

In a pcap, look for:

IP fragments.
Missing fragment sequences.
Reassembly failure.
Large UDP packets.
DNS or RTP payloads that exceed path limits.
ICMP packet-too-big messages.

If only one fragment is missing, the whole packet cannot be reassembled.

TLS handshake stalls and MTU

TLS handshakes can expose MTU issues because certificate messages may be larger than small TCP packets. A client may send ClientHello, the server may respond with certificate data, and then the handshake stalls because large packets disappear.

Users often search for "TLS handshake timeout MTU" or "HTTPS works on some sites not others" in this situation. The packet evidence may show repeated retransmission of the same large TLS-carrying TCP segment.

The problem is not the certificate itself; it is that larger packets cannot traverse the path.

VPN and tunnel overhead

VPNs reduce effective MTU because they wrap packets with additional headers. WireGuard, IPsec, OpenVPN, GRE, VXLAN, PPPoE, and cloud overlays all add overhead.

If the endpoint sends 1500-byte packets into a tunnel that cannot carry them, the tunnel must fragment, drop, or signal a smaller path MTU. If ICMP is blocked, the sender may never adapt.

MSS clamping at the VPN boundary is a common mitigation for TCP. UDP protocols may need application-level packet sizing or tunnel MTU adjustment.

Capture point and offload

Host captures may show packets larger than the actual wire packets because of segmentation offload. This can confuse MTU analysis. A 64 KB packet in a host capture may not have gone on the wire as one packet.

To avoid mistakes:

Inspect SYN MSS values.
Capture at a network tap, router, or tunnel boundary when possible.
Be aware of TSO/GSO/GRO/LRO offload.
Compare sender-side and receiver-side captures.
Look for retransmission behavior and ICMP feedback.

Do not diagnose MTU only from one apparent packet size in a host capture.

Checklist for MTU black hole PCAP analysis

Use this process:

Identify the affected conversation.
Inspect SYN and SYN-ACK MSS values.
Note whether a VPN, tunnel, PPPoE, or overlay is in the path.
Find the first stall or retransmission burst.
Check packet sizes around the stall.
Look for ICMP fragmentation-needed or packet-too-big messages.
Check whether the sender reduces segment size after ICMP.
Look for repeated retransmission of similar large segments.
Compare small and large requests.
Account for offload and capture point.

Final diagnosis

MTU black holes are size-dependent network failures. They often look like random application timeouts because small packets work and large packets fail. The strongest packet evidence is MSS negotiation, large segment retransmissions, missing ICMP packet-too-big feedback, fragmentation behavior, and path context such as VPN or tunnel overhead.

PCAP Surgery helps preserve the exact packet sequence needed to prove whether a stalled connection is really an MTU problem instead of generic packet loss, server slowness, or application failure.

Size-boundary test matrix

Run authorized requests with increasing payload sizes through the same path. Keep endpoint, protocol, connection reuse, and tunnel state fixed. Record the largest successful size and smallest repeatedly failing size, then include IP and transport overhead when relating application bytes to path MTU.

Test result	Likely lead	Required confirmation
Small succeeds, large stalls	MTU/PMTUD candidate	repeated size boundary and packet evidence
IPv4 fragments arrive	Fragmentation is occurring	whether every fragment reaches destination
DF packet followed by ICMP type 3 code 4	PMTUD feedback observed	sender reduces size appropriately
IPv6 Packet Too Big observed	Router reports smaller MTU	endpoint accepts and adapts
Large segment repeats, no ICMP visible	Possible black hole	second capture point/firewall policy
MSS already below path limit	Basic MSS mismatch less likely	inspect later encapsulation and direction

Host captures with TSO/GSO can show segments larger than the wire MTU. A server-side 64 KB “packet” may be a pre-segmentation buffer, not an Ethernet frame. Validate at a tap, receiving endpoint, or interface with offload understood before using displayed frame size as wire proof.

What is the difference between MSS and MTU?

MTU limits the IP packet on a path; TCP MSS advertises the maximum TCP payload a peer expects to receive. Header options, IPv6, VPN, GRE, VXLAN, PPPoE, or IPsec overhead reduce effective payload. An MSS value can be correct at the endpoint yet become too large after later encapsulation.

Does blocked ping prove an MTU black hole?

No. Echo may be blocked while ICMP Packet Too Big or Fragmentation Needed is allowed, or the reverse. Look for size-dependent application behavior and protocol-specific ICMP feedback rather than treating all ICMP as one policy.

Why can one direction fail?

Paths, encapsulation, MSS negotiation, and firewall handling can be asymmetric. Test uploads and downloads separately and name which sender repeatedly emits the unrecovered large range.

Actionable MTU handoff

Include capture point, tunnel/overlay layers, negotiated MSS in both directions, first failing packet size, DF or IPv6 behavior, ICMP direction and quoted packet, retransmission timing, successful control size, and interface counters. State when offload prevents an exact wire-size claim.

Retain the SYN/SYN-ACK, successful small transfer, failing large sequence, ICMP if present, and eventual timeout or recovery. Use capture scope help and the evidence preparation workflow so the derivative preserves both the negotiation and failure boundary.

Verify an MTU fix without masking the cause

Test one change at a time: allow required ICMP, reduce tunnel MTU, clamp TCP MSS, or correct encapsulation. A smaller application payload may hide the symptom without repairing PMTUD for other protocols. Repeat both the failing size and a boundary sweep, and confirm the packet sequence changes as predicted.

Fix under test	Expected packet evidence
ICMP policy corrected	Packet Too Big/Fragmentation Needed reaches sender
Endpoint PMTUD corrected	Sender reduces size after feedback
MSS clamping corrected	SYN negotiation advertises safe payload
Tunnel MTU corrected	Encapsulated packets cross without repeat stalls

Also test both directions, fresh connections, and IPv4/IPv6 where supported. Record the configuration owner and rollback. A successful browser request alone is not sufficient if it reused a connection negotiated before the change or happened to return a smaller response.

The final claim should name the tested path and size range: “Through the VPN path, 1,420-byte TCP payloads now complete after MSS clamping to X; the previous repeated large-range retransmissions are absent.” Avoid claiming a universal network MTU from one application flow.

Related guides

Continue with the same-language pages below. They cover adjacent stages without changing the canonical owner of this topic: