TCP Keepalive and Idle Timeout PCAP Analysis: Firewalls, NAT, Load Balancers, and Long-Lived Connections

An SSH session that freezes after exactly 15 minutes, a WebSocket that drops after an hour, or an idle database connection that returns “broken pipe” points to a timer more than random loss. The investigation should measure the last application data, TCP keepalive probes and ACKs, and the first FIN, RST, or silent retransmission to distinguish endpoint keepalive policy from a firewall, NAT, or load-balancer idle timeout. One capture establishes timing and visible packet direction, but a silent middlebox still needs its own logs or a second capture for attribution.

PCAP Surgery is useful because idle-timeout investigations depend on timing. You need the last real data packet, any TCP keepalive probes, ACKs, FIN/RST packets, and the exact idle duration.

What TCP keepalive is

TCP keepalive is an optional mechanism that sends small probes on an idle connection to check whether the peer is still reachable. It can also keep NAT and firewall state alive if probes occur more frequently than the middlebox timeout.

But defaults are often too slow for modern infrastructure. A firewall may expire idle state after 60 seconds while OS TCP keepalive may start much later.

Idle timeout symptoms

Common symptoms include:

• Connection works, then fails after a fixed idle time.
• First request after idle gets reset.
• WebSocket disconnects after exactly 60 seconds.
• Database pool has stale connections.
• SSH freezes through NAT.
• Load balancer sends RST after timeout.
• Client sends data after idle and receives no response.

Exact timing is the clue.

FIN vs RST vs silent drop

Middleboxes and endpoints can close idle connections in different ways:

• FIN: graceful close.
• RST: abortive close.
• Silent drop: no packet; later traffic is ignored.

If a firewall silently drops state, both endpoints may think the connection still exists. The next data packet triggers retransmissions or reset behavior.

Keepalive evidence

In a trace, look for small packets during idle periods. TCP keepalive probes often use sequence numbers just before the next expected byte. An analyzer may label them as keepalive.

Questions:

• Were keepalives sent?
• How often?
• Did the peer ACK them?
• Did a middlebox reset after a keepalive?
• Did probes start too late?
• Did the connection die before keepalive interval?

Load balancers and proxies

Load balancers often enforce idle timeouts. If the client expects a connection to survive 30 minutes but the load balancer closes idle connections after 60 seconds, the application must send heartbeats or reconnect.

Packet evidence can show who sent the close or reset and how long after last data.

Checklist

Use this workflow:

1. Identify the long-lived TCP connection.
2. Mark the last application data packet.
3. Measure idle time before failure.
4. Look for TCP keepalive probes.
5. Check whether probes are ACKed.
6. Identify FIN or RST sender.
7. If no close appears, look for silent drop and retransmissions.
8. Compare timeout to firewall/load-balancer settings.
9. Preserve timing when trimming.
10. Correlate with application heartbeats.

Final diagnosis

TCP idle failures are timing problems. The packet evidence can distinguish endpoint close, firewall/NAT state expiry, load balancer timeout, missing keepalive, too-slow keepalive, and silent drop.

PCAP Surgery helps preserve the idle interval and close/reset evidence so long-lived connection failures can be explained precisely.

Build an idle-connection timeline

Start the timer at the last bidirectional application activity, not merely the last packet. ACK-only traffic may not refresh every device’s idle state, and different devices can define “activity” differently. Record application data, heartbeats, TCP keepalive probes, ACKs, FIN/RST, retransmissions, and the first failed post-idle write.

Boundary	Time to record	Diagnostic purpose
Last application data	T0	Candidate start of application-idle timer
Last packet in either direction	T1	Network-level idle reference
First keepalive/heartbeat	T2	Whether policy begins before expiry
Last successful probe ACK	T3	Last proven bidirectional reachability
First FIN/RST or silent loss	T4	Failure boundary
First application error	T5	User-visible delay after network failure

Compare multiple connections. A repeatable T4−T0 near 60, 300, 900, or 3600 seconds strongly suggests a configured lifecycle boundary, but the value alone does not identify which device.

Distinguish TCP keepalive from application heartbeat

TCP keepalive is managed by the socket stack and may be invisible to application protocol logic. An application heartbeat—WebSocket Ping/Pong, database validation query, gRPC HTTP/2 PING, SSH traffic, or custom message—also proves that the application path is responsive.

Mechanism	Strength	Limitation
TCP keepalive probe/ACK	Tests TCP path and may refresh state	Does not prove application worker health
Application heartbeat	Tests protocol handling	Adds app traffic and must handle failure
Pool validation query	Detects stale DB connection before checkout	Adds latency/load
Reconnect-on-use	Simple recovery	First request after idle pays failure delay

Choose a policy that is shorter than the narrowest known middlebox timeout with margin, but avoid aggressive probes on millions of connections without capacity testing.

Do pure ACKs refresh firewall state?

It depends on implementation and policy. Some devices refresh on any packet, some on data or valid state transitions, and some proxy the connection. Verify the specific product configuration and observe timer/counter behavior.

Can NAT expiry create a reset?

After mapping expires, later traffic may be silently dropped, mapped differently, or answered with RST/ICMP by another node. Compare inside and outside captures and retain the translated tuple.

Why does the first request after idle fail?

Both endpoints may retain socket state while a middlebox has forgotten it. The application writes on a stale connection, retransmits, and eventually errors. Connection pools should validate, heartbeat, or discard connections before the infrastructure boundary.

Load balancer and proxy legs

A load balancer terminates separate client and backend TCP flows, each with its own idle timer. Client traffic may keep the frontend alive while the backend pool connection expires, or vice versa. Record last activity and closure on both legs and correlate by request ID.

Frontend	Backend	Likely behavior
Alive	Expired	First request triggers backend reconnect/error
Expired	Alive/pool retained	Client reconnect needed
Both drain by FIN	Graceful lifecycle policy
One leg RST at fixed age	Device/application abortive timeout lead

Silent-drop diagnosis with two captures

When no close packet appears, capture near both endpoints or on each side of the suspected device. If post-idle data leaves the client but never reaches the server while the device’s state table expired, the loss boundary is supported. Check capture drops and asymmetric routing before blaming the device.

If the server receives post-idle data and responds but the client does not see it, investigate the return path or expired reverse/NAT state. Packet presence across boundaries is stronger than a retransmission screenshot.

Verify a timeout or heartbeat fix

Test idle periods just below and above the former threshold, normal traffic, failover, and long-duration operation. Confirm heartbeats are sent and acknowledged, state counters refresh as expected, stale pooled connections are not handed to callers, and packet/CPU/battery costs remain acceptable.

Do not only raise infrastructure timeouts. That may retain dead state and increase resource usage. Match the intended connection lifecycle: explicit application heartbeat, pool max lifetime, graceful reconnect, or coordinated proxy timeout.

Actionable report

Include topology and TCP legs, flow tuples, capture points, last app data, last packet, keepalive/heartbeat schedule, ACK results, close/reset sender, retransmission sequence, exact idle duration across samples, NAT translation, and configured timers. State which timer remains unproven.

Preserve the handshake, last activity before idle, the complete idle interval, probe/heartbeat packets, failure, and reconnect. Use capture scope help and the TCP RST guide for abortive closes.

Direct answer: how do you prove a firewall idle timeout in PCAP?

Show a repeatable fixed interval from last qualifying activity to loss, verify endpoint keepalives/heartbeats, and compare captures before and after the firewall. A PCAP at one endpoint proves timing and missing replies; firewall logs/state counters or a second capture prove the device boundary.

Worked WebSocket example

Suppose a WebSocket exchanges data successfully, then becomes quiet. At 60.0 seconds after the last application frame, the load balancer sends FIN; the client acknowledges and reconnects. No TCP keepalive or WebSocket Ping occurs before the boundary. Across five connections, the interval varies by less than one second. This supports a configured graceful idle close.

If instead no FIN/RST is visible and the client’s next data at 75 seconds retransmits without reply, state expiration or silent path loss is a stronger lead. Capture on both sides of the load balancer. If the backend sees nothing and its state table no longer contains the flow, the boundary becomes actionable.

TCP keepalive versus WebSocket Ping: which is better?

They answer different questions. TCP keepalive validates connection-path reachability. WebSocket Ping/Pong validates the protocol endpoint and can integrate with application lifecycle. Many deployments use application heartbeat plus bounded reconnect because it detects a live TCP stack with a stalled application.

Should heartbeat interval equal the timeout?

No. Schedule with margin for jitter, loss, event-loop delay, and timer granularity. A heartbeat sent at exactly 60 seconds cannot reliably refresh a 60-second device timeout. Test worst-case load and avoid synchronized fleets that create heartbeat spikes.

Connection-pool QA

For database or HTTP pools, record pool max lifetime, idle validation, borrow-time validation, infrastructure timeout, retry policy, and whether failed requests are idempotent. Test a connection checked out immediately before and after the boundary.

Pool behavior	Expected safe result
Connection exceeds max lifetime	Retired before checkout
Idle validation fails	Closed and replaced
First write gets reset	Retry only when operation is safe
Backend drain starts	Pool stops assigning draining connection

Monitor reconnect rate and backend load after shortening lifetimes. A policy that prevents stale sockets but causes connection storms is not complete.

Can TLS session resumption affect reconnect evidence?

Yes. Reconnect may be faster with resumed TLS, while the underlying TCP flow is still new. Distinguish socket survival from session resumption and record fresh versus resumed handshakes.

The final fix should coordinate application heartbeat, pool lifetime, proxy/load-balancer settings, firewall/NAT state, and failure recovery rather than optimizing one timer in isolation.

<!-- multilingual-related-reading:start -->

Related guides

Continue with the same-language pages below. They cover adjacent stages without changing the canonical owner of this topic:

• TCP CLOSEWAIT and FINWAIT PCAP Analysis: Finding Connection Leaks, Half-Closes, and Shutdown Bugs
• TCP CWR Flag and ECN PCAP Analysis: Diagnose CE, ECE, and Congestion Without Packet Loss
• TCP MSS Clamping and VPN PCAP Analysis: Finding Oversized Segments, MTU Mismatch, and Slow Tunnels

<!-- multilingual-related-reading:end -->