Design: a BoringSSL-free QUIC-TLS interface for NIOSSL

[ydnar]

Design: a BoringSSL-free QUIC-TLS interface for NIOSSL

Status. F0 (API + bridge), F1 (handshake tests), and the delegate minimization (#4: alerts folded into the error, flush dropped) have landed on this fork's main, with follow-on hardening: #6 consumes the pending alert, #7 fails the secret callback on a null cipher, #8 tests the fatal-alert path, and #9 folds post-handshake processing into advance(). F2 has started: hostname verification and SNI land in #11 (issue #10). This issue is the living design doc and reflects the current API. Appendix comments below record prior art and the rationale for specific decisions. Nothing is committed upstream yet.

Motivation

QUIC (RFC 9001) does not run TLS over TCP records. It drives the TLS 1.3 handshake state machine directly and consumes its outputs per encryption level (Initial, 0-RTT, Handshake, 1-RTT): the handshake bytes to put in CRYPTO frames, and the read/write traffic secrets used to derive packet-protection keys. BoringSSL exposes this through its QUIC API (SSL_set_quic_method, SSL_provide_quic_data, SSL_process_quic_post_handshake, the SSL_QUIC_METHOD callbacks). swift-nio-ssl vendors a BoringSSL that has this compiled in, but NIOSSL surfaces none of it.

alta/swift-nio-quic, a pure-Swift QUIC transport on SwiftNIO, needs a TLS 1.3 handshake driver. This adds a thin, safe Swift surface over the BoringSSL QUIC API. (QUIC was part of the original motivation for moving swift-nio-ssl to BoringSSL.)

Hard constraints

1. No BoringSSL in the public API. Per the maintainer's policy (#484), the public surface contains no part of BoringSSL — it has no API/ABI stability. The API uses only Swift / NIOSSL types; the cipher suite is the IANA UInt16, translated from SSL_CIPHER * inside the module. No ssl_encryption_level_t, SSL_CIPHER, OpaquePointer, or CNIOBoringSSL import in any public declaration.
2. Minimal, additive, complete. Smallest surface that fully drives a QUIC handshake. New code under Sources/NIOSSL/QUIC/; edits to existing files surgical.
3. Convention-exact. API/doc style, Sendable correctness, availability, NIOSSLError, the platform matrix, and XCTest. Reuse NIOSSLContext / TLSConfiguration for certificates, keys, trust, verification, ALPN, SNI.
4. Security and correctness. Traffic secrets are copied out of callbacks immediately; never logged. Behavior matches BoringSSL's documented QUIC contract. Adversarial and boundary cases tested.

Current public API

/// A TLS encryption level, as used by QUIC (RFC 9001 §2.1).
public enum NIOTLSEncryptionLevel: Sendable, Hashable {
    case initial, earlyData, handshake, application
}

/// Whether the handshake acts as client or server.
public enum NIOSSLQUICRole: Sendable, Hashable { case client, server }

/// An error raised by a handshake.
public enum NIOSSLQUICError: Error, Hashable, Sendable {
    /// A fatal TLS alert (RFC 8446 §6); the QUIC layer maps it to
    /// CONNECTION_CLOSE 0x0100|alert (RFC 9001 §4.8).
    case tlsAlert(UInt8)
}

/// Receives the handshake's outputs. Invoked synchronously from
/// `provideHandshakeData(_:)` / `advance()`. There is no flush callback (flights
/// are bounded by the call that produces them) and no alert callback (alerts are
/// fatal and surface as a thrown NIOSSLQUICError).
public protocol NIOSSLQUICDelegate: AnyObject {
    func setReadSecret(level: NIOTLSEncryptionLevel, cipherSuite: UInt16, secret: [UInt8])
    func setWriteSecret(level: NIOTLSEncryptionLevel, cipherSuite: UInt16, secret: [UInt8])
    func writeHandshakeData(level: NIOTLSEncryptionLevel, _ data: [UInt8])
}

/// Drives a single QUIC TLS 1.3 handshake. Created from a configured
/// NIOSSLContext, so certificate/verification/ALPN/SNI config is reused.
/// Not thread-safe; use from one execution context (a connection's event loop).
public final class NIOSSLQUICHandshake {
    public enum State: Sendable, Hashable { case wantsMoreData, complete }

    public init(
        context: NIOSSLContext,
        role: NIOSSLQUICRole,
        serverHostname: String? = nil,
        localTransportParameters: [UInt8],
        delegate: any NIOSSLQUICDelegate
    ) throws

    /// Feed peer CRYPTO bytes at `level` (SSL_provide_quic_data).
    public func provideHandshakeData(level: NIOTLSEncryptionLevel, _ data: ByteBuffer) throws

    /// Advance the handshake (SSL_do_handshake), driving the delegate. After
    /// completion, drains buffered post-handshake messages (NewSessionTicket,
    /// KeyUpdate) via SSL_process_quic_post_handshake (#9). Throws
    /// NIOSSLQUICError on a fatal alert, else NIOSSLError.handshakeFailed.
    @discardableResult public func advance() throws -> State

    public var peerTransportParameters: [UInt8]? { get }
    public var negotiatedProtocol: String? { get }
}

Handshake bytes are fed in as ByteBuffer (zero-copy) and handed out as [UInt8] (copied out of BoringSSL-owned memory). Secrets are [UInt8].

Internal mapping to BoringSSL (behind the wall)

Public element	BoringSSL
init	SSL_new from the context's SSL_CTX; SSL_set_quic_method; SSL_set_connect/accept_state; SSL_set_quic_transport_params; SSL_set_ex_data for the trampolines
init (serverHostname, client)	SSL_set_tlsext_host_name (SNI); SSL_set1_host under .fullVerification, after validateSNIServerName (#11)
provideHandshakeData	SSL_provide_quic_data
advance	SSL_do_handshake while in init, else SSL_process_quic_post_handshake (#9); WANT_READ/WANT_WRITE → .wantsMoreData, success → .complete
peerTransportParameters	SSL_get_peer_quic_transport_params
negotiatedProtocol	SSL_get0_alpn_selected
setReadSecret/setWriteSecret	set_read_secret/set_write_secret; cipher via SSL_CIPHER_get_protocol_id; fails on a null cipher (#7)
writeHandshakeData	add_handshake_data
NIOSSLQUICError.tlsAlert	send_alert (captured, consumed, thrown from the next step)
(none)	flush_flight — internal no-op

The five C callbacks are non-capturing function pointers in one static SSL_QUIC_METHOD; each recovers self via SSL_get_ex_data + Unmanaged.fromOpaque (the pattern SSLConnection uses).

Edits to existing files

• SSLContext.createQUICSSLHandle() — an internal accessor returning a bare SSL (no BIO). The only change to existing code.

Resolved design questions

Full rationale and prior art are in the appendix comments; in brief:

1. Naming — NIOSSLQUICHandshake + provideHandshakeData (quiche names its BoringSSL wrapper Handshake and its feed provide_data; avoids collision with SSLConnection).
2. Cipher — IANA UInt16 (as Go, neqo); the only BoringSSL-free option that does not draw packet-protection key types into TLS.
3. Transport parameters — opaque [UInt8]; unanimous across stacks. TLS does not parse them.
4. Packaging — in-module under Sources/NIOSSL/QUIC/; a separate glue library (ngtcp2's model) is motivated by multi-backend support, which does not apply.
5. Alerts — folded into a thrown NIOSSLQUICError (no public alert precedent in NIOSSL; matches Go/neqo/s2n-quic).
6. Flush — none; an explicit flush is unique to BoringSSL and is a no-op in quiche and ngtcp2.
7. Push vs pull — delegate (push). This is a genuine industry split (push: quiche, OpenSSL 3.5; pull: Go, neqo, s2n-quic, msquic); justified here by single-event-loop ownership and the buffer-lifetime advantage. See Appendix A.

Deferred / known gaps

• Certificate verification on the QUIC path. Chain verification (trust, expiry, signature) is inherited from the context's SSL_CTX, so it already works under .fullVerification with trust roots configured — no custom callback needed. Hostname verification and SNI land in Verify the server hostname in the QUIC TLS handshake #11 via SSL_set1_host (issue Expose serverHostname on NIOSSLQUICHandshake for hostname verification #10). Still deferred: the user-supplied custom verify path (sslContextCallback / NIOSSLCustomVerificationCallback), which expects an SSLConnection the raw QUIC SSL does not have; and IP-address SAN matching (X509_VERIFY_PARAM_set1_ip), since SNI rejects IP literals.
• Transport parameters after the ClientHello. The constructor takes them up front, which cannot serve a server choosing them by SNI. An additive path is reserved (cf. Go's QUICTransportParametersRequired).
• 0-RTT / session tickets. Deferred to a later milestone; when added, model on Go's SendSessionTicket(EarlyData:Extra:) / QUICResumeSession / QUICStoreSession, including the opaque Extra ticket payload.
• ChaCha20 header protection (a swift-nio-quic, not NIOSSL, concern): swift-crypto exposes no raw ChaCha20 keystream; AES suites are fully covered by CryptoExtras.

Testing (XCTest, self-contained)

Drive a client and server NIOSSLQUICHandshake against each other in-process: feed each side's writeHandshakeData output to the other's provideHandshakeData (one flight at a time, advancing between — BoringSSL rejects data ahead of its read level), then assert both reach .complete, the read/write secrets agree peer-to-peer per level, a cipher suite is agreed, ALPN negotiates, and transport parameters round-trip. No sockets, no packets. Landed in F1.

Milestones

• F0 — types, delegate, NIOSSLQUICHandshake, SSL_QUIC_METHOD install. ✅ landed.
• F1 — handshake-driving + the in-process client/server test. ✅ landed.
• F2 — 🚧 in progress. Hostname verification + SNI landed (Verify the server hostname in the QUIC TLS handshake #11; chain verification was already inherited from the SSL_CTX). Post-handshake processing folded into advance() (Process post-handshake messages from advance() #9). Remaining: the user-supplied custom verify callback path; transport parameters after the ClientHello; (then) 0-RTT / session tickets.

Appendices (comments)

• Appendix A — Go crypto/tls prior art and the decisions it informed (alerts, flush, deferred transport parameters, push-vs-pull rationale).
• Appendix B — open-question resolutions with cross-implementation prior art and sources.

[ydnar]

Appendix A: Prior art (Go crypto/tls) and resulting decisions

Go added an equivalent TLS-for-QUIC API to its standard library. The design history is directly relevant and is a likely reference point in review, so it is summarized here along with the decisions it informs. These notes do not assert that the choices below are correct in general; they record rationale so that specific points can be contested.

A.1 References

• Proposal and discussion: golang/go#44886
• 0-RTT / session follow-ups: #60107, #63691; Go's own QUIC consumer: #58547
• Source: crypto/tls/quic.go
• Changelists: CL 493655 (accepted event API), CL 472621 (rejected synchronous-callback form), CL 496995 (0-RTT)
• Narrative: libp2p blog, 2023-09-13

A.2 Convergence

Both designs expose the same primitives, which appear to be dictated by RFC 9001: four encryption levels (initial / early / handshake / application); per-level read and write secrets plus cipher suite (IANA uint16); handshake bytes in and out; opaque transport-parameter exchange; post-handshake messages. Agreement across two unrelated TLS stacks suggests the primitive set is not the locus of design risk.

A.3 Divergence: push vs pull

Go drives TLS by draining an event queue (QUICConn.NextEvent). This proposal instead pushes via a delegate. Go reached its event model after building, then rejecting, a synchronous-callback form structurally equivalent to this one (CL 472621 → CL 493655). The recorded objections to callbacks were: (i) the callbacks only deliver values outward and were judged "indirect ways to deliver return values"; (ii) they oblige the caller to lock around state mutated inside an outstanding call; (iii) callback ordering is harder to reason about than a drained queue; (iv) events were considered easier to extend. The API's author dissented, preferring callbacks because a lent buffer needs no copy whereas events must copy handshake bytes.

This proposal retains the delegate form, with the following caveats stated plainly:

• Objections (i)–(iii) are weaker here than in a general-purpose library. A QUIC handshake object is owned by, and only invoked on, a single EventLoop; the delegate is called synchronously from advance() / provideHandshakeData(_:). There is no concurrent mutation to lock against, and the call sites bound the ordering. Delegation is also the prevailing ecosystem idiom.
• The buffer-lifetime point the Go author raised applies in our favor: a delegate can be handed bytes without the per-event copy the queue model requires.
• "Why not an event- or return-based API?" is expected in review; the above is offered as the answer, not as a claim of general superiority.

A.4 Decisions and reserved space

1. Transport parameters after the ClientHello. The constructor currently takes local transport parameters up front. That cannot serve a server choosing parameters after inspecting the ClientHello (e.g. by SNI). Room is reserved for an additive means of supplying transport parameters once the ClientHello is available, comparable to Go's QUICTransportParametersRequired. Not yet implemented; recorded so the initial shape does not preclude it.
2. Alerts. NIOSSL exposes no public TLS-alert type; handshake failures surface as NIOSSLError.handshakeFailed. Absent a precedent and preferring minimal surface, an outgoing alert should fold into the thrown error (carrying the alert code) rather than a distinct sendAlert delegate method, comparable to Go's AlertError. This removes one delegate method. To be applied; F0 currently retains sendAlert.
3. Flush. Go removed its flush callback on the grounds that an outgoing flight is bounded by one feeding call. Whether flushFlight() is necessary, or whether "flush after advance() returns" suffices, is to be re-evaluated.
4. 0-RTT and session tickets (later milestone). Should the API grow to cover these, Go's SendSessionTicket(EarlyData:Extra:), QUICResumeSession, and QUICStoreSession additions, including the opaque Extra ticket blob, are a tested template.

A.5 Guiding constraint

The fork exists to be accepted upstream. A smaller, more additive surface improves that prospect. Where in doubt this work will prefer fewer methods and fold capabilities into existing types (e.g. errors) rather than add new ones.

[ydnar]

Appendix B: Open-question resolutions

The §"Open questions" items are resolved as follows, grounded in a survey of how production stacks (BoringSSL SSL_QUIC_METHOD, ngtcp2/ngtcp2_crypto, Cloudflare quiche, OpenSSL 3.5 SSL_set_quic_tls_cbs, Go crypto/tls.QUICConn, Mozilla neqo/NSS, AWS s2n-quic, Microsoft msquic, Apple NWProtocolQUIC) handle the RFC 9001 §4 boundary. Citations at the end. These resolutions are provisional and open to revision.

B.1 Naming — keep NIOSSLQUICHandshake and provideHandshakeData

Handle names across stacks vary: Handshake (quiche), QUICConn (Go), Session (s2n-quic), Agent (neqo), SSL (BoringSSL/OpenSSL). quiche — also a BoringSSL consumer — names its wrapper Handshake and its inbound feed provide_data; BoringSSL's own verb is SSL_provide_quic_data. Our names therefore have direct precedent. …Connection is declined to avoid collision with NIOSSL's internal SSLConnection and with the QUIC transport connection.

B.2 Cipher suite — keep the IANA UInt16

The negotiated suite is surfaced three ways in the wild: raw IANA uint16 (Go, neqo); a library cipher object (SSL_CIPHER*, decoded by quiche and ngtcp2); or a pre-resolved AEAD/key object (msquic QUIC_PACKET_KEY, s2n-quic typed keys). For a public surface that must contain no BoringSSL types (see apple#484), the uint16 is the only option that neither leaks SSL_CIPHER nor draws packet-protection key types into the TLS layer. The QUIC layer maps the uint16 to (AEAD, hash); this is trivial and version-stable.

B.3 Transport parameters — opaque bytes

Unanimous: every surveyed stack carries transport parameters as the opaque quic_transport_parameters extension blob; Go's documentation states crypto/tls "does not parse or validate them." The structured §18 encoding is QUIC-version-specific and remains in the QUIC layer. (See A.4(1) for the reserved set-after-ClientHello path.)

B.4 0-RTT / session tickets — defer; reserve the ticket "extra" channel

Deferred to a later milestone. When implemented, the one mechanism every stack provides and that is easy to omit is a channel to stash app data (remembered transport parameters / flow-control limits) in the ticket, since 0-RTT must resume with consistent limits: Go SendSessionTicket(Extra:)/SessionState.Extra; ngtcp2 ngtcp2_conn_encode_0rtt_transport_params; quiche's combined session blob. Rejection signalling and a server early-data context are likewise required. The current API does not preclude these additively.

B.5 Packaging — in-module

Three philosophies exist: glue inside the TLS library (Go, OpenSSL 3.5), a separate per-backend glue library (ngtcp2_crypto), or glue inside the QUIC library (quiche, msquic, s2n-quic). The separate-library form is motivated by multi-backend support, which does not apply here (single backend: BoringSSL via NIOSSL). In-module under Sources/NIOSSL/QUIC/ keeps the one edit to existing code (createQUICSSLHandle()) internal.

B.6 Crypto layering — handshake here, packet protection in the QUIC layer

Universal split: the TLS library hands over only (secret, cipher suite); the QUIC layer performs RFC 9001 §5 HKDF-Expand-Label, AEAD, and header protection over a primitives library (quiche→ring, s2n-quic→aws-lc-rs, neqo→NSS, Go→stdlib). For the Swift consumer this is swift-crypto: HKDF, AES-GCM, and ChaCha20-Poly1305 are in Crypto; AES header protection is served by CryptoExtras AES.permute (single-block AES) or AES._CTR. The lone gap is a raw ChaCha20 keystream for ChaCha20 header protection, which swift-crypto's public API does not expose. This is a swift-nio-quic concern, not a NIOSSL-surface concern.

B.7 Reaffirmed (A.4)

• Flush. An explicit flush is unique to BoringSSL's SSL_QUIC_METHOD; quiche and ngtcp2, both BoringSSL consumers, implement flush_flight as a no-op because flights are bounded by the feeding call. flushFlight() is dropped from the delegate and handled internally as a no-op.
• Alerts. The pull-model stacks fold alerts into the returned error (Go AlertError, neqo CryptoAlert, s2n transport::Error); all map to CRYPTO_ERROR 0x0100–0x01ff. The outgoing alert is folded into the thrown error and sendAlert is removed.
• Push vs pull is a genuine split, not a settled practice: push/callback — quiche, OpenSSL 3.5; pull/poll — Go, neqo, s2n-quic, msquic; hybrid — ngtcp2. The delegate model is therefore mainstream; A.3 records the rationale for choosing it here.

Sources

Go: golang/go#44886, crypto/tls/quic.go. BoringSSL SSL_QUIC_METHOD: ssl.h. ngtcp2: programmers' guide, ngtcp2_crypto.h. quiche: tls/mod.rs. OpenSSL 3.5: SSL_set_quic_tls_cbs. neqo: neqo-transport/src/crypto.rs. s2n-quic: crypto/tls.rs. msquic: TLS.md, quic_tls.h. swift-crypto: HKDF, CryptoExtras/AES. Apple: NWProtocolQUIC.