2026-06-02

TCP MSS Clamping and VPN PCAP Analysis: Finding Oversized Segments, MTU Mismatch, and Slow Tunnels

How to analyze TCP MSS clamping problems in VPNs and tunnels, including SYN MSS values, MTU mismatch, oversized segments, retransmissions, fragmentation, and packet capture evidence.

If DNS, SSH, and small requests cross a VPN but file transfers or TLS records stall, compare the TCP MSS advertised before and after the tunnel boundary. Oversized repeated segments, absent ICMP feedback, and a payload-size threshold can reveal missing MSS clamping or a tunnel MTU mismatch. A capture shows the MSS option and packet sizes at its interface; offload and encapsulation mean it may not show the exact frame size on the constrained link.

MSS clamping is a common mitigation. It adjusts the TCP Maximum Segment Size advertised during the SYN exchange so endpoints avoid sending packets too large for the tunneled path.

PCAP Surgery is useful because the key evidence is in the SYN packets and the later retransmission pattern.

MSS and MTU relationship

MTU is the maximum packet size on a link. MSS is the maximum TCP payload size. On normal Ethernet with 1500-byte MTU, IPv4 TCP MSS is often 1460 bytes.

VPNs add overhead. If the tunnel reduces effective MTU but endpoints still advertise MSS 1460, packets can become too large after encapsulation.

What MSS clamping does

A router, firewall, or VPN gateway can rewrite TCP MSS in SYN packets:

Original MSS: 1460
Clamped MSS: 1360

This encourages endpoints to send smaller TCP segments that fit inside the tunnel.

If clamping is missing, too high, or applied only in one direction, large transfers may retransmit or stall.

Packet evidence

Inspect:

Client SYN MSS.
Server SYN-ACK MSS.
Whether a middlebox rewrote MSS.
Segment sizes after handshake.
Retransmissions of large segments.
ICMP packet-too-big or fragmentation-needed messages.
VPN/tunnel path and overhead.

If SYN MSS is 1460 across a tunnel that needs smaller packets, suspect missing clamping.

Asymmetric clamping

Sometimes one direction is clamped and the other is not. Downloads may work while uploads fail, or vice versa. Capture both directions and inspect both SYN and SYN-ACK.

Direction matters:

Client upload uses server-advertised MSS.
Server download uses client-advertised MSS.

Checklist

Use this workflow:

Identify the tunneled path.
Capture TCP SYN and SYN-ACK.
Record MSS values in both directions.
Estimate tunnel overhead and effective MTU.
Inspect large transfer segment sizes.
Look for repeated retransmissions.
Look for ICMP packet-too-big messages.
Compare inside and outside tunnel captures.
Test lower MSS or MTU as a controlled experiment.
Preserve handshake and failure packets together.

Final diagnosis

TCP MSS clamping problems are tunnel-size problems visible in packet captures. If MSS is too high for the VPN path, large segments retransmit, fragment, or disappear while small traffic works.

PCAP Surgery helps isolate the SYN/MSS evidence and the later transfer failure so VPN MTU problems can be proven instead of guessed.

Calculate a safe MSS from the actual tunnel

Start with the smallest IP MTU across the path and subtract inner IP and TCP headers. Then account for options and address family. MSS is TCP payload, not an Ethernet frame size.

IPv4 TCP MSS ≈ inner path MTU - 20-byte IPv4 header - 20-byte TCP header
IPv6 TCP MSS ≈ inner path MTU - 40-byte IPv6 header - 20-byte TCP header

TCP options increase header length on individual segments, but MSS negotiation is defined around the maximum segment data size. Tunnel overhead—IPsec, WireGuard, GRE, VXLAN, PPPoE, provider encapsulation—reduces the outer path capacity available to the inner packet.

Layer added	Size effect to inventory
PPPoE	Reduces common Ethernet IP MTU
GRE/IP-in-IP	Adds outer IP/tunnel headers
IPsec ESP	Adds variable headers, padding, authentication
WireGuard/UDP	Adds outer IP, UDP, and tunnel overhead
VXLAN/Geneve	Adds outer Ethernet/IP/UDP/tunnel headers
IPv6 outer path	Larger outer IP header than IPv4

Do not copy a universal clamp such as 1360 without calculating the deployed encapsulation and testing. An unnecessarily low MSS can reduce efficiency; an overly high value leaves the black hole.

Directional MSS worksheet

The MSS option advertises what the SYN sender can receive. Therefore, the client’s SYN MSS constrains server-to-client payload, and the server’s SYN-ACK MSS constrains client-to-server payload.

Handshake field	Controls	Capture both sides to check
Client SYN MSS	Server download segments	Was it rewritten entering/leaving VPN?
Server SYN-ACK MSS	Client upload segments	Is reverse-direction clamp applied?

This is why an upload-only failure can coexist with working downloads. Record original and post-clamp values at inside and outside capture points.

Can MSS be changed after the handshake?

The TCP MSS option is negotiated in SYN packets. Changing a firewall rule affects new connections, not an established connection’s negotiated values. Use fresh connections for validation and make connection reuse visible.

Does MSS clamping help UDP?

No. MSS is a TCP option. QUIC, RTP, DNS, and custom UDP require application sizing, PMTUD, or tunnel MTU handling. A “VPN fixed by MSS clamp” can still leave UDP broken.

Is fragmentation the same as clamping?

No. Clamping prevents TCP endpoints from creating oversized inner payloads. IPv4 fragmentation may split packets later; IPv6 routers do not fragment transit packets. Fragmentation adds loss sensitivity and may be filtered.

Inside/outside capture method

Capture the same new connection before encapsulation and after decapsulation or at both tunnel gateways. Match SYNs by tuple/NAT mapping, TCP timestamps, sequence, and time. Record:

Inner SYN/SYN-ACK MSS.
Any rewritten value at the boundary.
Outer tunnel type and observed size.
Inner TCP payload sizes.
ICMP Fragmentation Needed or Packet Too Big.
Retransmitted sequence ranges.
Capture offload state and dropped packets.

Host TSO/GSO can display segments larger than actual wire frames. Use a physical/tunnel boundary capture or receiver-side evidence before asserting the exact transmitted outer size.

Size-threshold test

Run authorized transfers with increasing payload sizes across fresh connections. Keep route, tunnel, endpoint, cipher, and direction constant. Record the largest consistently successful inner packet and the smallest failing size.

Result	Interpretation lead
Small TCP succeeds, larger TCP stalls	MTU/MSS candidate
Same large range retransmits	Feedback/adaptation failure
Packet Too Big reaches sender, size reduces	PMTUD working
Packet Too Big visible only at gateway	ICMP blocked on return path
Lower clamp fixes one direction	Asymmetric MSS policy
TCP fixed, UDP still fails	Non-TCP MTU handling remains

Use the MTU black-hole analysis for the broader PMTUD evidence and ICMP Packet Too Big guide for feedback interpretation.

Common configuration mistakes

Clamp applied to the wrong interface or chain

The rule may see outer encrypted traffic rather than inner SYNs, or only one routing direction. Confirm counters increment for the exact test connection and inspect MSS before/after.

Clamp affects only IPv4

IPv6 inner traffic requires its own policy and has different header/PMTUD behavior. Test both families rather than inferring from IPv4 success.

Double encapsulation

A corporate VPN inside a cloud overlay or mobile carrier tunnel adds more overhead than the gateway assumes. Inventory the complete path, including failover routes.

Path changes after handshake

If traffic migrates to a path with smaller MTU, the original MSS may no longer fit. PMTUD still needs to work; clamping is not a substitute for all path changes.

Validate the fix

Use fresh connections in both directions and repeat boundary-size tests. Require:

SYN and SYN-ACK show intended MSS after the relevant boundary.
Inner/outer packets fit the constrained path.
Prior repeated large-range retransmissions disappear.
ICMP feedback remains functional for future path changes.
TCP throughput and application success meet targets.
UDP/IPv6 are separately validated.
CPU, packet rate, and tunnel performance remain acceptable.

Do not claim success from a small ping or cached HTTP response. The test must cross the former size boundary.

Actionable report

Include topology, inner/outer address families, tunnel types, path MTUs, calculated overhead, capture points, client/server original and rewritten MSS, direction of failure, first failing payload/outer size, ICMP evidence, retransmission sequence, offload caveat, and exact rule/counter.

A bounded conclusion might be: “On new VPN connections, the client SYN retained MSS 1460 on the inner side, while the outer path supports an estimated inner MTU of 1380. Server-to-client payload ranges near 1460 repeatedly retransmitted and no Packet Too Big reached the server. Applying a client-direction MSS clamp to 1340 produced new SYNs with MSS 1340 and the controlled download completed.”

Preserve both handshakes, one successful small test, one failing large test, ICMP/retries, and the validated new flow. Use capture scope help and the packet preparation workflow for the handoff.

Direct answer: what should TCP MSS be for a VPN?

There is no universal number. Calculate the smallest supported inner MTU after all outer headers, subtract inner IP/TCP headers, apply the clamp in both required directions, and verify with fresh connections and packet-size evidence.

Monitoring after deployment

Track retransmissions by tunnel and direction, ICMP Packet Too Big/Fragmentation Needed, negotiated MSS samples, interface MTU/config drift, tunnel encapsulation changes, throughput, and connection failures by payload size. A cloud, carrier, or failover path change can invalidate an old clamp.

Alert when the advertised MSS exceeds the policy for new connections or when outer fragmentation/repeated large-range retransmission rises. Keep a synthetic large-transfer probe for every supported address family and tunnel route.

Should MSS clamping replace PMTUD?

No. It protects TCP handshake-negotiated payload on known paths, while PMTUD handles path changes and non-TCP traffic. Required ICMP feedback should still reach endpoints according to policy.

Can a clamp be too low?

Yes. Smaller segments increase packet rate, header overhead, CPU/interrupt work, and may reduce throughput. Choose the highest value proven safe across the intended path with margin for variable encapsulation, then monitor.

Why did only some servers fail?

Different servers may advertise different MSS, use IPv4 versus IPv6, sit behind another proxy, send larger TLS/application records, or follow a different tunnel route. Compare handshake, route, and size evidence instead of assuming one global VPN behavior.

Retest whenever tunnel encapsulation or failover routing changes.

Related guides

Continue with the same-language pages below. They cover adjacent stages without changing the canonical owner of this topic: