2026-06-02

TCP MSS Clamping and VPN PCAP Analysis: Finding Oversized Segments, MTU Mismatch, and Slow Tunnels

How to analyze TCP MSS clamping problems in VPNs and tunnels, including SYN MSS values, MTU mismatch, oversized segments, retransmissions, fragmentation, and packet capture evidence.

tcp mss clamping, vpn mtu, mtu mismatch, pcap analysis, retransmission, tunnel performance, path mtu

VPN and tunnel performance problems often look like random slowness. Small requests work. SSH connects. DNS works. Then file transfers stall, websites hang, TLS handshakes fail, or uploads crawl. Users search for "TCP MSS clamping VPN", "VPN MTU packet capture", "MSS mismatch pcap", "oversized TCP segments retransmission", and "slow tunnel MTU problem" when the path works for small packets but fails for larger traffic.

MSS clamping is a common mitigation. It adjusts the TCP Maximum Segment Size advertised during the SYN exchange so endpoints avoid sending packets too large for the tunneled path.

PCAP Surgery is useful because the key evidence is in the SYN packets and the later retransmission pattern.

MSS and MTU relationship

MTU is the maximum packet size on a link. MSS is the maximum TCP payload size. On normal Ethernet with 1500-byte MTU, IPv4 TCP MSS is often 1460 bytes.

VPNs add overhead. If the tunnel reduces effective MTU but endpoints still advertise MSS 1460, packets can become too large after encapsulation.

What MSS clamping does

A router, firewall, or VPN gateway can rewrite TCP MSS in SYN packets:

Original MSS: 1460
Clamped MSS: 1360

This encourages endpoints to send smaller TCP segments that fit inside the tunnel.

If clamping is missing, too high, or applied only in one direction, large transfers may retransmit or stall.

Packet evidence

Inspect:

  • Client SYN MSS.
  • Server SYN-ACK MSS.
  • Whether a middlebox rewrote MSS.
  • Segment sizes after handshake.
  • Retransmissions of large segments.
  • ICMP packet-too-big or fragmentation-needed messages.
  • VPN/tunnel path and overhead.

If SYN MSS is 1460 across a tunnel that needs smaller packets, suspect missing clamping.

Asymmetric clamping

Sometimes one direction is clamped and the other is not. Downloads may work while uploads fail, or vice versa. Capture both directions and inspect both SYN and SYN-ACK.

Direction matters:

  • Client upload uses server-advertised MSS.
  • Server download uses client-advertised MSS.

Checklist

Use this workflow:

  1. Identify the tunneled path.
  2. Capture TCP SYN and SYN-ACK.
  3. Record MSS values in both directions.
  4. Estimate tunnel overhead and effective MTU.
  5. Inspect large transfer segment sizes.
  6. Look for repeated retransmissions.
  7. Look for ICMP packet-too-big messages.
  8. Compare inside and outside tunnel captures.
  9. Test lower MSS or MTU as a controlled experiment.
  10. Preserve handshake and failure packets together.

Final diagnosis

TCP MSS clamping problems are tunnel-size problems visible in packet captures. If MSS is too high for the VPN path, large segments retransmit, fragment, or disappear while small traffic works.

PCAP Surgery helps isolate the SYN/MSS evidence and the later transfer failure so VPN MTU problems can be proven instead of guessed.