TLS Handshake Failure PCAP Analysis: ClientHello, ServerHello, Alerts, Resets, and Evidence

A TLS handshake failure PCAP should first answer one narrow question: what was the last protocol event that the capture actually observed? Find the TCP three-way handshake, the client's ClientHello, any ServerHello or TLS alert, and the first FIN or RST. That sequence can establish a useful failure boundary: before TLS began, after the client offered TLS, after the server chose parameters, or after certificate validation started. It cannot by itself prove every server-side policy decision, decrypt TLS traffic, or prove what happened at an uncaptured proxy hop.

That distinction makes a packet capture more useful than a generic “SSL error” message. An application may report the same message for an expired certificate, an SNI routing error, a server that rejects the offered protocol, a load balancer reset, or a path that drops a large handshake packet. The evidence in the PCAP should narrow the next check instead of assigning blame too early.

This guide explains a repeatable TLS handshake failure PCAP analysis workflow for client, server, support, and network teams. It uses the words TLS and SSL where people search for them, but the investigation concerns modern TLS handshakes. The examples apply whether the capture came from a browser, API client, embedded device, reverse proxy, or service-to-service connection.

What a packet capture can—and cannot—prove

A PCAP is an observation from one capture point. It can preserve packet order, endpoint addresses, TCP flags, selected TLS handshake metadata, alerts, and timing at that point. It does not automatically include server logs, certificate-store state, load-balancer policy, DNS resolver history, or packets lost before the capture interface saw them.

PCAP observation	A defensible conclusion	What it does not prove on its own
No SYN/ACK follows the client's SYN	The capture did not observe a completed TCP connection to that destination.	That the remote application is down; routing, firewall policy, asymmetric capture, or a different path may be involved.
TCP completes and the client sends ClientHello, but no server reply is observed	TLS started from the client's perspective and the observed conversation stopped after the offer.	That the server silently dropped it; a middlebox, return path, or capture gap may explain the absence.
A server TLS alert follows ClientHello	The server side of the captured connection rejected or terminated the handshake at that visible boundary.	The exact configuration rule without correlating server, proxy, or TLS-library logs.
A client alert follows certificate delivery	The client rejected what it observed during validation or negotiation.	Whether the reason was hostname, trust chain, expiry, local policy, clock, pinning, or another client rule.
A TCP RST appears instead of a TLS alert	A TCP endpoint or a device able to inject a reset terminated the observed flow.	Which process or appliance generated the reset without capture context and logs.

This evidence-first wording is important for incident handoffs. It is more useful to say “the client capture shows ClientHello at 10:03:14.121 and no return packet for 15 seconds” than “the server dropped TLS.” The first statement is reproducible; the second is a hypothesis to test.

Begin below TLS: confirm that TCP really works

Before reading any TLS field, make sure the selected packets belong to one client IP, server IP, port, and TCP stream. A TLS-looking application error can occur when TCP never becomes usable.

The expected start is simple:

1. Client sends SYN.
2. Server (or a device answering for it) sends SYN, ACK.
3. Client sends ACK.
4. Client sends application data carrying a TLS ClientHello.

If steps 1–3 do not complete, investigate reachability, firewall policy, port selection, NAT, route symmetry, or capture placement before calling it a TLS failure. A separate TCP SYN retransmission and no SYN-ACK analysis is the better workflow for that boundary.

If TCP completes but the client never sends a ClientHello, the failure may be local to the application, proxy configuration, name resolution, or a connection pool. The network trace can still help, but it has not observed a TLS negotiation yet.

Read the handshake as a timeline, not as isolated packets

TLS diagnosis is usually clearer when every packet is placed in order. A filter that shows only “TLS” can accidentally hide TCP resets, retransmissions, DNS context, or the delay that explains the application timeout. Start broad enough to preserve the sequence, then create a focused evidence file.

Stage	What to inspect in the PCAP	Why it matters	Sensible next check
DNS, if captured	Requested hostname, returned address, response timing	Confirms which name and IP the client was likely using.	Compare the intended hostname and chosen address; do not assume DNS was captured.
TCP setup	SYN, SYN/ACK, ACK, MSS, retransmissions, resets	Separates reachability from TLS negotiation.	Check route, firewall, NAT, and capture point if setup is incomplete.
ClientHello	Offered TLS versions, cipher suites, SNI, ALPN, supported groups, signature algorithms	Records what the client actually offered.	Compare to the server or proxy policy.
ServerHello	Chosen version and cipher suite; timing after ClientHello	Shows the server accepted at least part of the proposal.	Continue to certificate, alert, or encrypted-handshake boundary.
Certificate/validation phase	Certificate records when visible; client alert or close	May locate the rejection after server authentication began.	Check name, chain, expiry, local trust, clock, and proxy behavior.
Alert, FIN, or RST	Sender, direction, alert description when visible, timing	Establishes how the visible flow ended.	Correlate with logs at the sender or intervening proxy.

For a support case, note packet numbers and timestamps for the first ClientHello, first ServerHello, first alert, and first TCP close. Those four markers let another engineer reproduce the conclusion without scanning a large capture from scratch.

ClientHello: the client's visible offer

The ClientHello is often the most useful packet in a TLS handshake failure PCAP. Depending on protocol version and capture decoding, it can expose the parameters the client offered before encryption protects more of the exchange:

• TLS protocol versions it supports;
• cipher suites;
• Server Name Indication (SNI), when present;
• ALPN application protocols such as HTTP/2 or HTTP/1.1;
• supported groups and key-share information;
• signature algorithms; and
• extensions that can matter to a proxy, load balancer, or older TLS endpoint.

Do not infer a fault merely because an unfamiliar value appears. The practical question is whether the offering is compatible with the endpoint that received it. An old device might offer only TLS versions or cipher suites disabled by policy. A client connecting by IP address might omit SNI. A modern application could offer correct settings while a proxy routes the request to an unexpected virtual host.

SNI and hostname are early routing evidence

SNI lets a TLS client tell a shared endpoint which hostname it intends to reach before HTTP begins. That matters because the HTTP Host header is sent later, inside TLS. A client can reach the correct IP address but still receive a default certificate or the wrong backend if its SNI is absent or wrong.

Use the dedicated TLS SNI mismatch PCAP analysis when the evidence points to a wrong certificate, name mismatch, reverse-proxy route, or IP-based connection. Capture both legs if a proxy terminates TLS: the client-to-proxy handshake can be correct while the proxy-to-upstream handshake fails for a different SNI value.

ALPN is negotiation evidence, not a diagnosis by itself

ALPN indicates the application protocol preferences a client offered during TLS. If the server chooses an unexpected protocol, or no compatible protocol exists, that can matter for an HTTP/2 or HTTP/1.1 failure. It is not evidence that TLS itself failed unless the timeline and visible alert support that conclusion. For that narrower problem, see TLS ALPN and HTTP/2 PCAP analysis.

ServerHello, certificates, and the TLS 1.3 visibility boundary

If the server responds with ServerHello, the capture has observed a meaningful transition: the server accepted enough of the client's offer to choose handshake parameters. That does not mean the connection is ready for application data. Certificate validation, client authentication, policy checks, and later encrypted handshake messages can still fail.

TLS version matters when interpreting what remains visible. In TLS 1.2, more handshake messages are often directly identifiable in a normal capture, including certificate-related records. In TLS 1.3, much of the handshake after ServerHello is encrypted. A decoder may still identify record-level context, but it cannot reveal every certificate or policy detail without the appropriate lawful decryption material. Never promise that a PCAP tool can “read all TLS content” just because it displays a TLS stream.

That creates a safe diagnostic boundary:

• ClientHello followed by ServerHello means the server selected parameters.
• A client alert shortly after the certificate-related stage is evidence that the client terminated the visible negotiation.
• An application log saying “certificate verify failed” can identify the reason more precisely than the PCAP alone.
• The absence of readable certificate details in TLS 1.3 is expected, not proof that the capture is corrupt.

For an expiry-specific question, use the expired TLS certificate PCAP workflow alongside certificate inventory and client logs. Packet timing can show where validation stopped; it cannot reliably determine the device clock or trust-store contents.

TLS alerts: keep their direction and timing

TLS alerts are structured handshake evidence. When the alert description is visible, record its level, description, sender, and the packet immediately before it. Common labels include handshake_failure, protocol_version, unknown_ca, bad_certificate, certificate_expired, illegal_parameter, and close_notify.

Alert descriptions should guide a hypothesis, not end the investigation. For example:

Visible sequence	Strongest evidence-based reading	Checks that should follow
ClientHello → server fatal protocol_version	The observed server endpoint rejected the offered version at negotiation.	Server/proxy TLS minimum and the client's actual offered versions.
ClientHello → server fatal handshake_failure	The server ended the visible handshake after the client offer.	Cipher, group, signature, SNI, client-certificate, or policy compatibility; obtain logs before choosing one.
ServerHello/certificate stage → client fatal unknown_ca	The client rejected the visible server authentication path.	Client trust store, intermediate chain delivery, enterprise interception, and capture point.
ServerHello/certificate stage → client close or reset with no readable alert	The client side closed the observed flow during or after authentication.	Client library error, pinning, hostname, expiry, local clock, and TLS version visibility.
ClientHello → no TLS alert → TCP RST	The observed TCP flow was reset after the offer.	Endpoint logs, firewall/load-balancer behavior, injected reset possibilities, and a second capture point.

Avoid treating close_notify as a fatal failure by default. It is a normal TLS closure alert. The timeline and the application result determine whether it is relevant to the reported problem.

A reset is not the same as a TLS rejection

One of the most common errors in SSL troubleshooting is to label every reset as a certificate failure. A TCP RST is a TCP-layer event. It may come from the server host, an application listener, a proxy, a firewall, a NAT device, or an injected device on the path. In some environments it can even be missing at the client capture point because the reset traveled on a different path.

When a reset follows ClientHello, preserve these facts:

• the stream's client and server endpoints;
• TCP sequence and acknowledgement context;
• whether data was acknowledged before the RST;
• the exact delay from ClientHello to reset;
• whether a TLS alert appeared first; and
• whether the capture occurred on the client, proxy, server, or a network tap.

This supports an honest next action: inspect the logs and policies at the system that can see that boundary. It does not support claiming “the remote server reset it” merely because the packet's source address is the server address; a proxy or network device can use that address in the observed path.

Capture placement changes the conclusion

When client and server teams disagree, take the capture point seriously. A packet missing from a client-side PCAP means “not observed at the client interface,” not necessarily “never sent.” A load balancer can complete TCP on behalf of an origin. A reverse proxy can create two independent TLS handshakes. Asymmetric routing can show only one direction at a tap.

Capture point	Particularly useful for	Main limitation
Client host	Actual client offer, local alert/reset, resolver and route context	Cannot show policy or packets that never return to the client.
Edge firewall or load balancer	Client-facing decisions, TLS termination, NAT and policy events	May not reveal the proxy-to-origin connection.
Reverse proxy	Separating inbound and upstream TLS legs	Requires clear timestamps and interfaces to avoid mixing streams.
Origin server	Arrival at the service and local response	Does not prove what happened before the final hop.
SPAN/tap	Broad path context	May have drops, missing directions, or offload artifacts.

If the trace ends after ClientHello, capture on both sides of the suspected middlebox before changing certificates or cipher settings. Two short, synchronized captures often resolve the uncertainty faster than a much larger trace from only one end.

Check retransmissions, segmentation, and timeouts before changing TLS policy

Large ClientHello or certificate messages can span TCP segments. Do not assume that the absence of a decoded TLS field means the message was not sent; verify whether the capture contains all stream segments and whether the analyser has reassembled them. Also distinguish a TLS handshake timeout from a TCP retransmission pattern.

Useful questions include:

1. Was the ClientHello acknowledged?
2. Did the client retransmit the same TCP payload?
3. Did the server return anything before the application deadline?
4. Is there a gap consistent with a firewall idle timeout, loss, or application timeout?
5. Does the same transaction work on a different network path or with a smaller/updated client?

TCP offload and capture loss can make packet sizes, checksums, or ordering look unusual at a host interface. Treat those as capture-quality clues, not as automatic proof of a TLS protocol fault. Record the operating system, interface, capture tool, and whether hardware offload was active when a result will be handed to another team.

Build a defensible TLS evidence file

The original capture may be too large or too sensitive to send broadly. The goal is not to delete every detail; it is to preserve the smallest context that still supports the observed handshake boundary. For a TLS handshake failure, that normally means a short time window around DNS (if relevant), TCP setup, the handshake records, alerts/resets, and a few seconds of idle time before and after.

PCAP Surgery is a local desktop workbench for opening classic PCAP and basic PCAPNG files, examining decoded packet evidence, narrowing a time range or conversation, applying controlled redaction, repairing checksums where needed, and exporting a smaller focused capture. It is not a TLS decryption service and should not be used to imply that encrypted application payloads have become readable.

A practical workflow is:

1. Keep an untouched original with its acquisition details.
2. Open the local file and identify the five-tuple and time window that contain the reported attempt.
3. Review TCP setup, ClientHello, response records, alerts, FINs, resets, and retransmissions in sequence.
4. Keep any DNS packets and proxy-side leg that are necessary to explain the hostname or route.
5. Redact only fields that are not required to establish the timing and protocol evidence.
6. Export the focused capture with a short evidence note listing packet numbers, timestamps, capture point, and the conclusion's uncertainty.

The capture-scope guide can help define the right time window, while the download page provides the current local desktop build. Do not send a trimmed file as a replacement for the original when incident retention rules require both.

Handoff checklist for support, security, and application teams

The following table turns a vague “TLS fails” ticket into a reviewable packet-analysis handoff.

Include	Why the next team needs it
User-visible error, client/library version, and exact URL hostname	Connects the packet evidence to the initiating request without guessing the intended name.
Capture point and interface	States what the PCAP could and could not observe.
Client/server endpoints, port, and first packet timestamp	Lets another engineer find the same stream.
Packet numbers for SYN, ClientHello, ServerHello, alert, FIN, and RST	Preserves the evidence boundary even if packet decoding differs.
A short timeline with elapsed milliseconds	Reveals whether the symptom is immediate policy rejection, loss, or timeout-like delay.
SNI, offered TLS versions, and ALPN when visible	Supplies compatible-negotiation facts without exposing unrelated payload.
The exact statement of uncertainty	Prevents “missing from this capture” from being translated into “never happened.”
Relevant proxy/server/client log correlation	Provides the policy or validation reason the PCAP alone cannot prove.

Frequently asked questions

Does a ClientHello with no ServerHello prove the server dropped the handshake?

No. It proves the selected capture did not observe a reply after the ClientHello. The cause could be server policy, a proxy, a firewall, a return-path issue, packet loss, a capture gap, or a different TLS leg. Check a second capture point and the relevant logs before assigning the cause.

Can I identify a bad certificate from a TLS handshake PCAP?

Sometimes the capture shows certificate metadata or a client alert after the certificate stage, which is strong evidence about where validation ended. It does not reliably identify every reason for rejection—such as local trust-store state, pinning, hostname policy, or device time—without client and server context. See the focused expired certificate guide for the right evidence set.

Why can I see ClientHello but not the later certificate details?

TLS 1.3 encrypts much of the handshake after ServerHello. That is normal protocol behavior. A capture parser can still show boundaries and timing, but it cannot disclose all encrypted values unless authorized decryption keys and a suitable capture are available.

Is a TCP reset after ClientHello an SSL handshake failure?

It is a connection termination observed during the TLS attempt, not necessarily a TLS alert. Report the reset, direction, timing, and capture point; then correlate with endpoint, proxy, or firewall logs. Calling it a certificate failure without that evidence is not defensible.

What should I send to a vendor without exposing the whole capture?

Retain the original securely, then share a focused, reviewed capture that includes the TCP and TLS timeline necessary to reproduce the boundary. Preserve the packet numbers, timestamps, SNI/ALPN only when needed, alert/reset context, and a note about redactions. A PCAP Surgery workflow is designed for this sort of controlled evidence handoff.

The useful conclusion from TLS packet evidence

A good TLS handshake failure analysis does not promise more than the trace proves. It identifies the last visible handshake event, distinguishes TCP transport evidence from TLS alert evidence, notes the capture point, and turns the boundary into a precise next check. That is enough to move a support case from “SSL is broken” to a testable question: did the server reject the offer, did the client reject authentication, did a middlebox reset the flow, or did the capture miss the other side of the transaction?

When that answer is documented with timestamps, packet numbers, a short table, and the relevant PCAP Surgery help, the next team receives a useful technical record instead of a speculative diagnosis.

<!-- multilingual-related-reading:start -->

Related guides

Continue with the same-language pages below. They cover adjacent stages without changing the canonical owner of this topic:

• TLS Certificate and Handshake Failure PCAP Analysis: Expired Certs, Alerts, SNI, and Connection Resets
• TLS SNI Mismatch PCAP Analysis: Wrong Certificate, Wrong Host, Proxy Routing, and Handshake Failure
• SNI vs ALPN: TLS Handshake Analysis and HTTP/2 Negotiation in Wireshark PCAPs

<!-- multilingual-related-reading:end -->