2026-06-02

TLS Certificate and Handshake Failure PCAP Analysis: Expired Certs, Alerts, SNI, and Connection Resets

How to analyze TLS handshake failures in packet captures, including expired certificates, unknown CA, SNI mismatch, TLS alerts, ClientHello, ServerHello, and TCP resets.

For “TLS Certificate and Handshake Failure PCAP Analysis: Expired Certs, Alerts, SNI, and Connection Resets”, the short answer is: How to analyze TLS handshake failures in packet captures, including expired certificates, unknown CA, SNI mismatch, TLS alerts, ClientHello, ServerHello, and TCP resets. Treat that statement as a result to verify, not as a promise that every input, device, project, or environment behaves identically. A complete result records the starting state, the exact action, the visible output, and the condition that proves the task is finished in PCAP Surgery.

“SSL handshake failed,” “unknown CA,” “certificate expired,” and “connection reset” describe different points in TLS setup but are often collapsed into one application error. Follow TCP establishment, ClientHello fields such as SNI and ALPN, ServerHello, certificate visibility, TLS alerts, and the first FIN/RST to locate the break in the handshake. Modern encryption and missing session keys limit certificate and alert interpretation, yet the capture still establishes protocol order, endpoints, timing, and whether TCP failed first. PCAP Surgery is useful because TLS failure evidence often needs to be isolated carefully. You may need to trim a large trace to one connection while preserving DNS lookup, TCP handshake, TLS handshake, alerts, resets, and timing.

TLS handshake stages

A simplified TLS handshake includes:

TCP SYN
TCP SYN-ACK
TCP ACK
TLS ClientHello
TLS ServerHello
Certificate and key exchange messages
TLS Finished
Application data

Depending on TLS version, some details are encrypted earlier than others. TLS 1.3 hides more handshake content than older TLS versions, but the first steps still reveal useful metadata.

The first question is: did the connection fail before TLS, during TLS, or after TLS?

Failure before TLS

If the TCP connection does not establish, the problem is not a certificate problem. Look for:

SYN retransmissions.
TCP reset after SYN.
No route or firewall drop.
Proxy refusal.
Server port closed.

An application may still report a generic SSL error if it expected TLS but never established a TCP path. The pcap separates network reachability from TLS negotiation.

ClientHello evidence

The ClientHello can reveal:

Server Name Indication, or SNI.
ALPN protocols such as h2 or http/1.1.
TLS versions offered.
Cipher suites offered.
Supported groups and extensions.

SNI is especially important. A server hosting multiple certificates needs SNI to choose the right certificate. If the client omits SNI or sends the wrong hostname, the server may present a default certificate that does not match the requested site.

This creates errors like:

Certificate name mismatch.
Wrong certificate served.
Handshake failure.
Server closes connection after ClientHello.

Expired certificate vs unknown CA

From the application perspective, certificate failures may look similar. But they mean different things:

Expired certificate: the certificate validity period is over.
Unknown CA: the certificate chain is not trusted.
Name mismatch: the certificate does not match the hostname.
Incomplete chain: intermediate certificate missing.
Revoked certificate: certificate is no longer valid.

In packet captures, certificate details may be visible in TLS 1.2 handshakes if not encrypted by the negotiated mode. In many modern cases, application logs are needed for exact certificate validation reason. The pcap still shows whether the server presented handshake data and whether the client responded with an alert or reset.

TLS alerts

TLS alerts are extremely useful. Common alerts include:

unknown_ca
bad_certificate
certificate_expired
handshake_failure
protocol_version
unrecognized_name
decrypt_error
close_notify

If the client sends an alert after receiving certificate data, the server may not be the side that "failed." The client rejected something.

If the server sends handshake_failure immediately after ClientHello, the server may not support the offered TLS version, cipher suite, SNI, or client certificate behavior.

Connection reset after ClientHello

A TCP reset after ClientHello can mean:

Middlebox blocked the handshake.
Server rejected TLS version or SNI.
Proxy expected plain HTTP, not TLS.
Client connected to the wrong port.
Server process crashed or closed aggressively.
Security appliance terminated the flow.

Do not assume every reset is a certificate problem. Check who sent the reset and when.

SNI mismatch

SNI mismatch is common in environments with load balancers, reverse proxies, Kubernetes ingress, CDNs, and multi-tenant servers. The TCP destination IP may host many domains. The server chooses certificate and backend based on SNI.

If the client connects by IP address, uses an internal hostname, or sends the wrong SNI, the server may present the wrong certificate.

Packet evidence:

ClientHello SNI value.
Destination IP and port.
DNS answer used by the client.
Server certificate or alert behavior.

If DNS resolves correctly but SNI is wrong, the application or proxy configuration is the place to inspect.

Protocol version and cipher mismatch

Modern clients may reject old TLS versions. Old servers may not support modern versions or cipher suites. The pcap can show what the client offered and how the server responded.

Look for:

Client offers only TLS 1.2/1.3, server requires older behavior.
Server sends protocol version alert.
Server closes without ServerHello.
Client and server share no cipher suite.
Middlebox interferes with TLS 1.3.

This is common in legacy embedded devices, old cameras, outdated appliances, and internal services that have not been updated.

DNS, TCP, and TLS timing together

For user-visible latency, preserve the whole path:

DNS query and response.
TCP connect.
TLS handshake.
Application data or failure.

A "TLS timeout" may actually be DNS delay, TCP retransmission, MTU black hole during certificate transfer, or server delay before ServerHello.

PCAP Surgery helps keep this timeline intact when reducing a trace.

Checklist for TLS handshake PCAP analysis

Use this process:

Identify the target host, IP, and port.
Preserve DNS lookup before the connection.
Confirm TCP handshake succeeds.
Inspect ClientHello for SNI, ALPN, versions, and ciphers.
Check whether ServerHello appears.
Look for certificate messages where visible.
Look for TLS alerts and identify sender.
Look for TCP resets and identify sender.
Measure timing between handshake steps.
Correlate with application certificate validation logs.

Final diagnosis

TLS certificate and handshake failures are not one category. The packet trace can distinguish TCP reachability failure, ClientHello rejection, SNI mismatch, protocol version mismatch, certificate validation failure, TLS alert, TCP reset, MTU-related stall, and server delay.

PCAP Surgery helps by preserving the DNS, TCP, TLS, alert, reset, and timing evidence needed to explain why a secure connection failed without reducing the issue to a generic "SSL error."

TLS failure-stage matrix

Locate the last completed stage before assigning a certificate cause. A TCP reset before ClientHello cannot be an expired-certificate decision by that client. A TLS alert after certificate delivery is stronger, but application logs are still needed to map encrypted or generic alerts to a validation reason.

Last observed stage	Packet-level lead	Next corroboration
SYN retries, no handshake	TCP/path/reachability	firewall, listener, route counters
ClientHello, no ServerHello	server/middlebox rejection or loss	server TLS logs and second capture
ServerHello then alert	negotiated TLS failure	alert direction, endpoint logs
Certificate then client alert/FIN	client validation lead	trust-store and hostname logs
Handshake succeeds, app data resets	post-handshake/application issue	HTTP/app/proxy telemetry

Can a PCAP prove a certificate is expired?

If the certificate is visible and its validity interval can be decoded, the trace can show the presented certificate dates. Proving that expiry caused the client failure also requires the client’s validation time, clock, policy, chain, and log or alert evidence.

How does SNI mismatch appear?

The ClientHello may show the requested SNI, and the server may present a certificate or choose a virtual host that does not match. Compare requested name, returned certificate names where visible, destination address, and server configuration. A missing SNI from an old client is a separate case.

What about TLS 1.3 encryption?

More handshake content becomes encrypted after ServerHello. Without session keys, preserve the visible ClientHello, negotiated progression, record directions/sizes, alert or reset boundary, and timing. State which certificate or alert fields were unavailable rather than guessing.

Could MTU cause a TLS handshake stall?

Yes. ClientHello may succeed while a larger certificate flight repeatedly disappears. Look for size-dependent retransmissions and ICMP behavior, then apply the MTU black-hole workflow.

TLS incident handoff

Provide hostname, DNS answer, client/server 4-tuple, capture point, TLS version offer, SNI, ALPN, visible certificate identity/dates, last completed stage, alert or RST direction, elapsed gaps, and matching endpoint log window. Preserve DNS through termination in the focused artifact.

Use capture scope help and the packet preparation workflow before sharing. TLS metadata can identify hosts and organizations even when payload is encrypted, so privacy review remains required.

Certificate validation checklist outside the PCAP

Packet chronology should direct a client-side certificate check, not replace it. Export or inspect the presented chain through an authorized method and compare hostname, SAN entries, validity interval, issuer, intermediates, key usage, revocation policy, and client clock. Record which trust store and application runtime performed validation.

Validation item	Common failure
Subject Alternative Name	Requested hostname is absent
Not Before / Not After	Client clock falls outside validity
Intermediate chain	Server omits required intermediate
Trust anchor	Client store lacks or distrusts issuer
Key usage / policy	Certificate not valid for server auth
SNI routing	Server presents another virtual host’s chain

Different clients can produce different outcomes because OS stores, embedded runtimes, pinned certificates, and proxy interception differ. A browser success does not prove an IoT client or Java runtime will accept the same chain.

What should a regression test cover?

Test a fresh connection with correct hostname and SNI, supported protocol/cipher, expected ALPN, clean trust store, and controlled clock. Include negative cases for expired, hostname-mismatched, and missing-intermediate chains only in an authorized lab. Capture and preserve the alert/reset direction and matching application error.

After a fix, require both successful handshake progression and the expected client validation log. A changed reset pattern without application success may simply move the failure later.

Safe final wording for a TLS case

Separate the visible stage from the validation diagnosis. Example: “TCP completed and the client sent a TLS 1.3 ClientHello with SNI camera.example. The server replied through ServerHello; the client then closed after the encrypted handshake flight. The capture establishes direction and timing but does not expose the client’s certificate-validation reason. Client logs show hostname mismatch.” This joins packet evidence and host evidence without pretending the encrypted trace alone proved the policy decision.

If the certificate itself is visible, quote fingerprints and validity facts in the case record rather than copying private keys or unrelated session material. Keep time synchronization uncertainty explicit when expiry is close to a boundary. One minute of clock error can change the outcome near Not Before or Not After.

Retest from every materially different client trust stack affected by the incident, not only from the analyst workstation.

Related guides

Continue with the same-language pages below. They cover adjacent stages without changing the canonical owner of this topic: