RTSP Digest Authentication Debugging

When an RTSP client answers a Digest challenge and receives another 401 Unauthorized, the protocol evidence is specific enough to test. Digest is not “the password, but encrypted.” It is a response computed from several exact strings. A mismatch in any input can look identical to a wrong secret at the player UI.

This guide isolates that calculation. For URL parsing, account permissions, protected stream paths, and method-by-method triage, begin with the broader RTSP authentication-loop guide.

Preserve the challenge and retry as one evidence unit

A simplified exchange looks like this:

C → S  DESCRIBE rtsp://camera.example/stream RTSP/1.0
       CSeq: 7

S → C  RTSP/1.0 401 Unauthorized
       CSeq: 7
       WWW-Authenticate: Digest realm="camera",
                         nonce="server-value",
                         qop="auth"

C → S  DESCRIBE rtsp://camera.example/stream RTSP/1.0
       CSeq: 8
       Authorization: Digest username="viewer", ...

Keep the request method and URI with both messages. A detached WWW-Authenticate header is insufficient because the method and digest URI are inputs to the response.

Reconstruct the calculation without logging the password

For the common MD5-based form, define:

HA1 = MD5(username : realm : password)
HA2 = MD5(method : digest-uri)

When no qop value is used:

response = MD5(HA1 : nonce : HA2)

With qop=auth, the response also includes nonce count and client nonce:

response = MD5(
  HA1 : nonce : nc : cnonce : qop : HA2
)

Colons above are literal separators. Case, whitespace, URI form, method, quoted-field parsing, and hexadecimal representation all matter. Use the algorithm and parameters actually advertised by the server; do not assume that every camera implements the same variant.

Recompute only in a controlled diagnostic environment. Do not place the password, HA1, or a reusable authorization value in public logs. A redacted report can still preserve field names, the advertised algorithm, realm, nonce-change behavior, qop, nc, method, URI shape, and final status.

Compare the realm byte for byte

The realm participates in HA1. A displayed label that “looks the same” may still differ because of capitalization, whitespace, encoding, or a challenge cached from another endpoint.

Check whether:

• the retry copied the realm from the current challenge;
• a proxy or recorder produced a different realm from the camera;
• the client reused a challenge saved for another host or channel;
• a firmware or configuration change invalidated cached authentication state;
• multiple WWW-Authenticate challenges were offered and the client answered the intended one.

Do not normalize the realm for convenience. Hash the exact challenge value.

Treat the nonce as server-owned state

The nonce comes from the server and can expire or be replaced. If a new 401 includes stale=true and a fresh nonce, the server is indicating that the previous nonce is no longer acceptable; the client should form a new response using the new challenge.

Build a short nonce timeline:

Attempt	Challenge nonce	Retry nonce	stale	Result
1	fingerprint A	fingerprint A	absent	401 or success
2	fingerprint B	fingerprint A or B	true/absent	401 or success

Use fingerprints or redacted labels in a shared report. If the server issues nonce B while the client continues answering with nonce A, the cache boundary is visible. If the client uses B and still fails, move to the other inputs.

For qop=auth, verify that nc is formatted as the expected eight hexadecimal digits and advances appropriately for uses of the same nonce. Confirm that cnonce is present in both the calculation and header when required.

Compare three URI representations

Digest failures often hide in URI handling. Put these values side by side:

1. The URI on the RTSP request line.
2. The uri parameter inside Authorization.
3. The URI string used to compute HA2.

They must follow the server’s expected form. Potential differences include absolute RTSP URI versus path, explicit versus implicit port, percent-encoding, query order, case-sensitive path components, or a proxy-rewritten target.

Do not “fix” the trace by normalizing all three before comparison. The difference itself is the evidence.

Bind the response to the actual RTSP method

HA2 includes the method. A response computed for DESCRIBE cannot simply be copied onto SETUP or PLAY. When the first protected method succeeds and a later one fails, confirm that the client recalculated authorization with the later method and current URI.

Use CSeq to associate every challenge with the correct request, especially when several track SETUP exchanges occur close together. Also retain the RTSP session identifier separately; it is session state, not a substitute for a correct Digest response.

Interpret repeated 401 patterns

• No Authorization retry: scheme negotiation or credential configuration failed before Digest computation.
• Same challenge and same bad response: stable input mismatch; compare method, realm, URI, user, and secret.
• New nonce, old retry nonce: stale challenge cache.
• Correct-looking fields but one client fails: compare raw request-line URI and parameter formatting between clients.
• DESCRIBE succeeds, SETUP fails: recalculate for method and track URI; inspect per-method authorization policy.
• Digest succeeds, later status is not 401: stop debugging the hash and move to resource, session, transport, SDP, RTP, or codec evidence.

Capture a safe regression case

Once the failure is understood, retain a sanitized fixture containing request methods, URI shapes, CSeq, challenge parameter structure, redacted authorization metadata, and expected status transitions. Never retain working plaintext credentials in the fixture.

RTSP Inspector records the RTSP control timeline and redacts authorization evidence in its diagnostic path. The valuable output is not “login failed”; it is a reproducible boundary such as “client reused the previous nonce,” “Digest URI differed from the request URI,” or “authorization was not recalculated for SETUP.”