Skip to content

Verify the peer certificate chain in the application - #16

Merged
ydnar merged 1 commit into
mainfrom
quic-custom-cert-verify
Jun 19, 2026
Merged

Verify the peer certificate chain in the application#16
ydnar merged 1 commit into
mainfrom
quic-custom-cert-verify

Conversation

@ydnar

@ydnar ydnar commented Jun 18, 2026

Copy link
Copy Markdown

Motivation

QUIC drives the TLS 1.3 handshake directly rather than over the record layer (RFC 9001 § 4), and authenticating the peer's certificate is part of that handshake. BoringSSL's built-in verification — chain validation against the SSL_CTX trust roots, plus the SSL_set1_host name check from #11 — covers the common case. Some applications need the trust decision themselves, though: certificate pinning, a custom trust store, or validation that consults an external service.

BoringSSL exposes exactly this through SSL_set_custom_verify under SSL_VERIFY_PEER. The callback can return ssl_verify_retry to pause the handshake (SSL_ERROR_WANT_CERTIFICATE_VERIFY) while the application rules on the chain, then resume with ssl_verify_ok or ssl_verify_invalid. NIOSSLQUICHandshake had no way to reach this: verification was always the context's job.

This is the fork side of #13 — the sans-I/O pull two-step the design issue settled on. The swift-nio-quic consumer side (the async QUICConfiguration.verifyCertificateChain closure, the ConnectionCore park/resume, and the TransportHandler Task bridge) follows in a separate PR; this change builds the primitive it bridges onto.

Modifications

A new opt-in client init parameter, customCertificateVerification (default false), installs a custom-verify callback under SSL_VERIFY_PEER so it fires even when the context's mode is .none, and skips the built-in SSL_set1_host name check. The application then owns trust evaluation and the hostname check both; SNI is still sent so the server can select a certificate.

advance() returns a new State case, wantsCertificateVerify([NIOSSLCertificate]), carrying the DER-decoded peer chain when the handshake parks. resumeVerification(_:) takes the application's NIOSSLVerificationResult; the next advance() resumes the handshake or fails it with the certificate alert. The callback state machine mirrors the record path's CustomVerifyManager, minus the promise: stash the chain and park on first call, keep parking while a verdict is pending, report ssl_verify_ok / ssl_verify_invalid once one is recorded.

The former useServerHostname splits into sendServerNameIndication (SNI, sent whenever a hostname is present) and requireServerHostname (SSL_set1_host, the built-in name check), making the SNI-always / name-check-conditional split explicit. Peer-chain extraction is hoisted into a shared SSLConnection.peerCertificateChain(fromSSL:), reused by the record path and the QUIC handshake, which holds an SSL handle but no SSLConnection. The public surface stays NIOSSL-only — NIOSSLCertificate, NIOSSLVerificationResult, no BoringSSL types — so the upstream bar holds.

Validation: three XCTest cases in NIOSSLQUICHandshakeTests — a verifier owning trust and the name check completes against an untrusted, name-mismatched certificate and sees the peer chain; a .failed verdict fails the handshake with NIOSSLQUICError.tlsAlert; and the default path never invokes the verifier. The existing #11 SNI / SSL_set1_host cases are unchanged.

Result

A client QUIC handshake can verify the peer certificate chain in the application, replacing BoringSSL's built-in trust and name checks (RFC 9001 § 4).


Generated by Claude Code

Motivation:

QUIC drives the TLS 1.3 handshake directly rather than over the record
layer (RFC 9001 § 4), and authenticating the peer's certificate is part
of that handshake. BoringSSL's built-in verification — chain validation
against the SSL_CTX trust roots, plus the SSL_set1_host name check —
covers the common case. Some applications need the trust decision
themselves, though: certificate pinning, a custom trust store, or
validation that consults an external service.

BoringSSL exposes exactly this through SSL_set_custom_verify under
SSL_VERIFY_PEER. The callback can return ssl_verify_retry to pause the
handshake (SSL_ERROR_WANT_CERTIFICATE_VERIFY) while the application rules
on the chain, then resume with ssl_verify_ok or ssl_verify_invalid.
NIOSSLQUICHandshake had no way to reach this: verification was always the
context's job.

Modifications:

A new opt-in client init parameter, customCertificateVerification
(default false), installs a custom-verify callback under SSL_VERIFY_PEER
so it fires even when the context's mode is .none, and skips the built-in
SSL_set1_host name check. The application then owns trust evaluation and
the hostname check both; SNI is still sent so the server can select a
certificate.

advance() returns a new State case, wantsCertificateVerify, carrying the
DER-decoded peer chain when the handshake parks. resumeVerification(_:)
takes the application's NIOSSLVerificationResult; the next advance()
resumes the handshake or fails it with the certificate alert. The
callback state machine mirrors the record path's CustomVerifyManager,
minus the promise.

The former useServerHostname splits into sendServerNameIndication (SNI,
sent whenever a hostname is present) and requireServerHostname
(SSL_set1_host, the built-in name check), making the SNI-always /
name-check-conditional split explicit. Peer-chain extraction is hoisted
into a shared SSLConnection.peerCertificateChain(fromSSL:), reused by the
record path and the QUIC handshake, which holds an SSL handle but no
SSLConnection.

Validation: three XCTest cases — a verifier owning trust and the name
check completes against an untrusted, name-mismatched certificate and
sees the peer chain; a .failed verdict fails the handshake with
NIOSSLQUICError.tlsAlert; and the default path never invokes the verifier.

Result:

A client QUIC handshake can verify the peer certificate chain in the
application, replacing BoringSSL's built-in trust and name checks
(RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
@ydnar
ydnar marked this pull request as ready for review June 18, 2026 12:41
@ydnar
ydnar merged commit 8e18c29 into main Jun 19, 2026
44 of 45 checks passed
ydnar pushed a commit to alta/swift-nio-quic that referenced this pull request Jun 19, 2026
The custom certificate-verification API this PR consumes landed on the
fork's main (alta/swift-nio-ssl#16), so the dependency points back at
`main` and Package.resolved pins the squash-merge commit, dropping the
PR-branch pin used while the fork change was in review.
ydnar added a commit to alta/swift-nio-quic that referenced this pull request Jun 19, 2026
Motivation:

Authenticating the server's certificate is part of the QUIC TLS 1.3 handshake (RFC 9001 § 4), which QUIC drives directly rather than over the record layer. Built-in verification — the chain checked against TLSConfiguration.certificateVerification, plus the serverHostname name check (#109) — covers the common case. Some clients need the trust decision themselves: certificate pinning, a custom trust store, or validation that consults an external service such as an OCSP responder.

Such validation is network-bound and may suspend, while the QUIC connection core is sans-I/O and synchronous. The decision therefore belongs at the connection's edge, on the event loop, with the core pausing the handshake until a verdict arrives. The fork's NIOSSLQUICHandshake provides that pause as a sans-I/O two-step (alta/swift-nio-ssl#16): it parks at .wantsCertificateVerify carrying the peer chain and resumes on resumeVerification.

Modifications:

QUICConfiguration gains verifyTLS, an optional @sendable (QUICHandshakeContext) async throws -> NIOSSLVerificationResult. When set, it replaces the built-in trust and hostname checks — the application owns both — and opts the client handshake into the fork's custom verification. QUICHandshakeContext is an immutable snapshot of what the handshake has settled when the certificate is ruled on: the peer chain, the serverHostname dialed, the negotiated protocol and cipher suite, and the peer address — modeled on Go's VerifyConnection(ConnectionState), deliberately a value rather than the live QUICConnection.

The pause threads through three layers. TLSHandshake propagates the opt-in, maps the fork's .wantsCertificateVerify into its own State, and forwards the verdict; it also splits CRYPTO reassembly from advancing (provideCryptoFrame), so a flight re-sent while paused buffers without re-driving the handshake. ConnectionCore parks on the chain — surfaced once by takeCertificateVerificationRequest — and resumes on the verdict via resumeCertificateVerification, which on .failed fails the handshake with the certificate alert (CRYPTO_ERROR, § 4.8); while paused it buffers received CRYPTO and keeps ACKing, with no new timer, so a verdict that never comes lets the connection idle out. TransportHandler assembles the QUICHandshakeContext, runs the verifier in a detached Task off the loop, and hops the verdict back through a NIOLoopBound to resume and flush; the Task is cancelled on teardown.

A verifier that throws — a check that could not complete — currently fails the handshake the same as a .failed verdict; splitting that into an INTERNAL_ERROR close is a deferred refinement. The mTLS direction, a server custom-verifying the client's certificate, is a separate follow-up tracked on alta/swift-nio-ssl#13.

Validation: sans-I/O ConnectionCore tests drive a paused handshake to completion on a .certificateVerified verdict (asserting the verifier saw the server's leaf certificate, against an untrusted, name-mismatched server) and to a CRYPTO_ERROR close on .failed; loopback tests over real sockets prove the async path end to end — a verifier approving an otherwise-untrusted chain completes the dial (asserting the context carried the dialed name, the chain, and the negotiated ALPN), and a verifier rejecting it over a built-in .none policy fails the dial.

Result:

A QUIC client can verify the server's certificate chain in the application, given an immutable QUICHandshakeContext, replacing the built-in trust and name checks (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
ydnar added a commit that referenced this pull request Jun 20, 2026
Motivation:

Mutual authentication in the QUIC TLS 1.3 handshake (RFC 9001 § 4) has the server verify the client's certificate as well as the reverse. The client direction shipped in #16: customCertificateVerification installs SSL_set_custom_verify and surfaces the peer chain at .wantsCertificateVerify for a verdict. The server arm did nothing past SSL_set_accept_state, so a server could neither custom-verify a client nor, since the same call requests the client certificate, ask for one at all. And once it can, whether a client must present one (reject anonymous) is a separate policy.

Modifications:

customCertificateVerification on a server now installs the same custom-verify callback in the server arm. SSL_VERIFY_PEER makes the server send the CertificateRequest and routes the client's chain through the existing .wantsCertificateVerify / resumeVerification two-step; the State case, resumeVerification, and the callback's state machine are unchanged and role-agnostic.

Whether the client must present a certificate follows the context's certificateVerification, the same lever the record path uses: .fullVerification / .noHostnameVerification add SSL_VERIFY_FAIL_IF_NO_PEER_CERT (required), .none leaves it optional. installCustomVerify takes the CertificateVerification and reads only that bit from it; a client uses the default (.none), since a server presents its certificate unconditionally. Optional is optional in the strong sense: with no fail bit, BoringSSL does not consult the verifier when the client sends none, so the handshake completes and the verifier cannot reject the anonymous case; required fails the handshake before the callback. The public API is unchanged: only the server arm and the customCertificateVerification documentation move.

Validation: XCTest cases in NIOSSLQUICHandshakeTests cover a server verifying a client that presents a certificate (completing and seeing the chain); a .failed verdict failing with the certificate alert; an optional server completing with no client certificate, with the verifier not consulted; and a required (.fullVerification) server rejecting a client that presents none (failing before the verifier) and completing with one.

Result:

A server QUIC handshake can verify the client's certificate in the application, optional or required per certificateVerification: the mutual-TLS counterpart of the client-side verification (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
ydnar added a commit to alta/swift-nio-quic that referenced this pull request Jun 20, 2026
Motivation:

Mutual authentication in the QUIC TLS 1.3 handshake (RFC 9001 § 4) has the server verify the client's certificate, the reverse of the client authenticating the server. The client direction shipped in #133: QUICConfiguration.verifyTLS hands the application an immutable QUICHandshakeContext and awaits its verdict, parking the handshake through the fork's pause/resume two-step (alta/swift-nio-ssl#16). The hook and the park/resume are role-agnostic, but the consumer wired them only on the dial path, so a server never requested a client certificate or ran the verifier on one.

Modifications:

verifyTLS set on a server configuration now means "request the client's certificate and custom-verify it", symmetric with its client meaning. The flag threads through the accept path: TransportCore.Accepting carries it, the server's deferred handshake (built from the client's first Initial) is constructed with customCertificateVerification, and the fork's server arm (alta/swift-nio-ssl#17) installs the custom-verify callback, which is also what sends the CertificateRequest. The accepted connection's attachment receives the listener's verifier, and TransportHandler assembles the QUICHandshakeContext as it does for a client: peerCertificates is the client's chain, serverHostname is nil, since a server has no SNI to check.

Whether a client must present a certificate rides the existing tls.certificateVerification rather than a new knob, the same lever NIOSSL servers and HTTP/2 already use: .fullVerification and .noHostnameVerification reject a client that presents none, .none (the default) leaves it optional. That lever stands on its own. A server that sets certificateVerification and trustRoots gets working mutual TLS with no verifyTLS at all, the client chain validated against the configured roots exactly as a NIOSSL TLS server does. verifyTLS is for owning the trust decision in the application instead of delegating it to the trust store.

Validation: loopback tests drive a server whose verifyTLS approves the client's certificate (asserting the verifier saw the client's chain and a nil serverHostname) and a .fullVerification server that rejects a client presenting none (the handshake fails server-side, so the connection is never delivered); two more cover the built-in trust-store path with no verifier, accepting a trusted client and rejecting an anonymous one. The client-side and sans-I/O park/resume coverage from #133 is unchanged, the mechanism being the same code reached on the server.

Result:

A QUIC server can verify the client's certificate, optional or required via certificateVerification, either against its trust store or in the application through verifyTLS: the mutual-TLS counterpart of the client-side verification (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
ydnar added a commit that referenced this pull request Jun 30, 2026
Motivation:

QUIC drives the TLS 1.3 handshake directly rather than over the record layer (RFC 9001 § 4), and authenticating the peer's certificate is part of that handshake. BoringSSL's built-in verification — chain validation against the SSL_CTX trust roots plus the SSL_set1_host name check — covers the common case, but some applications need the trust decision themselves: certificate pinning, a custom trust store, or validation that consults an external service.

BoringSSL exposes this through SSL_set_custom_verify under SSL_VERIFY_PEER: the callback returns ssl_verify_retry to pause the handshake (SSL_ERROR_WANT_CERTIFICATE_VERIFY) while the application rules on the chain, then resumes with ssl_verify_ok or ssl_verify_invalid. NIOSSLQUICHandshake had no way to reach it.

Modifications:

A new opt-in client init parameter, customCertificateVerification (default false), installs a custom-verify callback under SSL_VERIFY_PEER and skips the built-in SSL_set1_host name check, so the application owns trust evaluation and the hostname check both; SNI is still sent so the server can select a certificate. advance() returns a new State case, wantsCertificateVerify([NIOSSLCertificate]), carrying the peer chain when the handshake parks; resumeVerification(_:) takes the verdict and the next advance() resumes the handshake or fails it with the certificate alert. The former useServerHostname splits into sendServerNameIndication and requireServerHostname, and peer-chain extraction is hoisted into a shared SSLConnection.peerCertificateChain(fromSSL:). The public surface stays NIOSSL-only — no BoringSSL types.

Validation: three XCTest cases in NIOSSLQUICHandshakeTests — a verifier owning trust and the name check completes against an untrusted, name-mismatched certificate and sees the chain; a .failed verdict fails with NIOSSLQUICError.tlsAlert; and the default path never invokes the verifier.

Result:

A client QUIC handshake can verify the peer certificate chain in the application, replacing BoringSSL's built-in trust and name checks (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
ydnar added a commit that referenced this pull request Jun 30, 2026
Motivation:

Mutual authentication in the QUIC TLS 1.3 handshake (RFC 9001 § 4) has the server verify the client's certificate as well as the reverse. The client direction shipped in #16: customCertificateVerification installs SSL_set_custom_verify and surfaces the peer chain at .wantsCertificateVerify for a verdict. The server arm did nothing past SSL_set_accept_state, so a server could neither custom-verify a client nor, since the same call requests the client certificate, ask for one at all. And once it can, whether a client must present one (reject anonymous) is a separate policy.

Modifications:

customCertificateVerification on a server now installs the same custom-verify callback in the server arm. SSL_VERIFY_PEER makes the server send the CertificateRequest and routes the client's chain through the existing .wantsCertificateVerify / resumeVerification two-step; the State case, resumeVerification, and the callback's state machine are unchanged and role-agnostic.

Whether the client must present a certificate follows the context's certificateVerification, the same lever the record path uses: .fullVerification / .noHostnameVerification add SSL_VERIFY_FAIL_IF_NO_PEER_CERT (required), .none leaves it optional. installCustomVerify takes the CertificateVerification and reads only that bit from it; a client uses the default (.none), since a server presents its certificate unconditionally. Optional is optional in the strong sense: with no fail bit, BoringSSL does not consult the verifier when the client sends none, so the handshake completes and the verifier cannot reject the anonymous case; required fails the handshake before the callback. The public API is unchanged: only the server arm and the customCertificateVerification documentation move.

Validation: XCTest cases in NIOSSLQUICHandshakeTests cover a server verifying a client that presents a certificate (completing and seeing the chain); a .failed verdict failing with the certificate alert; an optional server completing with no client certificate, with the verifier not consulted; and a required (.fullVerification) server rejecting a client that presents none (failing before the verifier) and completing with one.

Result:

A server QUIC handshake can verify the client's certificate in the application, optional or required per certificateVerification: the mutual-TLS counterpart of the client-side verification (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
ydnar added a commit that referenced this pull request Jul 17, 2026
Motivation:

QUIC drives the TLS 1.3 handshake directly rather than over the record layer (RFC 9001 § 4), and authenticating the peer's certificate is part of that handshake. BoringSSL's built-in verification — chain validation against the SSL_CTX trust roots plus the SSL_set1_host name check — covers the common case, but some applications need the trust decision themselves: certificate pinning, a custom trust store, or validation that consults an external service.

BoringSSL exposes this through SSL_set_custom_verify under SSL_VERIFY_PEER: the callback returns ssl_verify_retry to pause the handshake (SSL_ERROR_WANT_CERTIFICATE_VERIFY) while the application rules on the chain, then resumes with ssl_verify_ok or ssl_verify_invalid. NIOSSLQUICHandshake had no way to reach it.

Modifications:

A new opt-in client init parameter, customCertificateVerification (default false), installs a custom-verify callback under SSL_VERIFY_PEER and skips the built-in SSL_set1_host name check, so the application owns trust evaluation and the hostname check both; SNI is still sent so the server can select a certificate. advance() returns a new State case, wantsCertificateVerify([NIOSSLCertificate]), carrying the peer chain when the handshake parks; resumeVerification(_:) takes the verdict and the next advance() resumes the handshake or fails it with the certificate alert. The former useServerHostname splits into sendServerNameIndication and requireServerHostname, and peer-chain extraction is hoisted into a shared SSLConnection.peerCertificateChain(fromSSL:). The public surface stays NIOSSL-only — no BoringSSL types.

Validation: three XCTest cases in NIOSSLQUICHandshakeTests — a verifier owning trust and the name check completes against an untrusted, name-mismatched certificate and sees the chain; a .failed verdict fails with NIOSSLQUICError.tlsAlert; and the default path never invokes the verifier.

Result:

A client QUIC handshake can verify the peer certificate chain in the application, replacing BoringSSL's built-in trust and name checks (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
ydnar added a commit that referenced this pull request Jul 17, 2026
Motivation:

Mutual authentication in the QUIC TLS 1.3 handshake (RFC 9001 § 4) has the server verify the client's certificate as well as the reverse. The client direction shipped in #16: customCertificateVerification installs SSL_set_custom_verify and surfaces the peer chain at .wantsCertificateVerify for a verdict. The server arm did nothing past SSL_set_accept_state, so a server could neither custom-verify a client nor, since the same call requests the client certificate, ask for one at all. And once it can, whether a client must present one (reject anonymous) is a separate policy.

Modifications:

customCertificateVerification on a server now installs the same custom-verify callback in the server arm. SSL_VERIFY_PEER makes the server send the CertificateRequest and routes the client's chain through the existing .wantsCertificateVerify / resumeVerification two-step; the State case, resumeVerification, and the callback's state machine are unchanged and role-agnostic.

Whether the client must present a certificate follows the context's certificateVerification, the same lever the record path uses: .fullVerification / .noHostnameVerification add SSL_VERIFY_FAIL_IF_NO_PEER_CERT (required), .none leaves it optional. installCustomVerify takes the CertificateVerification and reads only that bit from it; a client uses the default (.none), since a server presents its certificate unconditionally. Optional is optional in the strong sense: with no fail bit, BoringSSL does not consult the verifier when the client sends none, so the handshake completes and the verifier cannot reject the anonymous case; required fails the handshake before the callback. The public API is unchanged: only the server arm and the customCertificateVerification documentation move.

Validation: XCTest cases in NIOSSLQUICHandshakeTests cover a server verifying a client that presents a certificate (completing and seeing the chain); a .failed verdict failing with the certificate alert; an optional server completing with no client certificate, with the verifier not consulted; and a required (.fullVerification) server rejecting a client that presents none (failing before the verifier) and completing with one.

Result:

A server QUIC handshake can verify the client's certificate in the application, optional or required per certificateVerification: the mutual-TLS counterpart of the client-side verification (RFC 9001 § 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cu2dW2zXuAnwDRaiHr7feR
@ydnar
ydnar deleted the quic-custom-cert-verify branch August 5, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants