ssl: TLS client aborts handshake with handshake_decode_error when CertificateRequest contains a CA name with countryName encoded as UTF8String

[sadjow]

Describe the bug

During a TLS 1.2 handshake, if the server's CertificateRequest message contains a certificate_authorities entry whose DN violates an X.520 type constraint (real-world example: countryName encoded as UTF8String instead of PrintableString), the client aborts the whole handshake with a fatal decode_error alert (handshake_decode_error). This happens even when no client certificate is configured, and regardless of verify mode or any other ssl option.

Root cause: ssl_handshake:decode_cert_auths/2 calls public_key:pkix_normalize_name/1 on every advertised CA name with no error handling:

decode_cert_auths(<<?UINT16(Len), Auth:Len/binary, Rest/binary>>, Acc) ->
    decode_cert_auths(Rest, [public_key:pkix_normalize_name(Auth) | Acc]).

pkix_normalize_name/1 raises on the non-conformant DN, the error propagates into the try in tls_handshake:get_tls_handshakes_aux/4, and becomes ?ALERT_REC(?FATAL, ?DECODE_ERROR, handshake_decode_error) (tls_handshake.erl:460 on maint-28).

The certificate_authorities list is advisory — a hint for client certificate selection (RFC 5246 §7.4.4). Other TLS implementations (OpenSSL/curl, Go, Java, .NET, browsers) treat these entries leniently or as opaque bytes and interoperate fine with such servers. As a result, Erlang/Elixir clients (raw ssl and everything built on it: httpc, Mint, Finch, hackney) cannot connect at all to affected servers. In practice these are typically IIS servers configured to advertise their machine trust store, which may contain badly-encoded third-party CA certificates that the server operator does not control either.

To Reproduce

Server-independent repro — the DN below is a real CA name harvested from a live server's CertificateRequest (its countryName is UTF8String-encoded, tag 12, where PrintableString, tag 19, is required):

1> Der = base64:decode("MHAxCzAJBgNVBAYMAkJSMRMwEQYDVQQKDApJQ1AtQnJhc2lsMTQwMgYDVQQLDCtBdXRvcmlkYWRlIENlcnRpZmljYWRvcmEgUmFpeiBCcmFzaWxlaXJhIHY1MRYwFAYDVQQDDA1BQyBTeW5ndWxhcklE").
2> public_key:pkix_normalize_name(Der).
** exception error: no match of right hand side value
   {error,{asn1,{{'Type not compatible with table constraint',
                  {error,{asn1,{wrong_tag,{{expected,19},{got,12,{12,"BR"}}}}}}},
                 [...]}}}

openssl asn1parse decodes the same DN without complaint: C=BR, O=ICP-Brasil, OU=Autoridade Certificadora Raiz Brasileira v5, CN=AC SyngularID (all attributes UTF8STRING).

Live handshake repro against a public server that currently exhibits this (the Brazilian national NFS-e homologation environment; its CertificateRequest advertises 223 CA names, one of which is the DN above — no client certificate needed):

1> ssl:connect("adn.producaorestrita.nfse.gov.br", 443,
               [{verify, verify_none}, {versions, ['tlsv1.2']}], 20000).
{error,{tls_alert,
        {decode_error,
         "TLS client: In state certify at tls_handshake.erl:460 generated CLIENT ALERT: Fatal - Decode Error\n handshake_decode_error"}}}

openssl s_client -connect adn.producaorestrita.nfse.gov.br:443 -servername adn.producaorestrita.nfse.gov.br -tls1_2 completes the handshake against the same server.

The failure is independent of verify, cacerts, depth, ciphers, signature_algs, and of whether certfile/keyfile are configured.

Expected behavior

Undecodable entries in the advisory certificate_authorities list should be skipped (optionally logged), not abort the handshake. That would match the tolerance the ssl/public_key apps already apply elsewhere:

• ssl_handshake:decode_sign_alg/2 silently ignores unknown signature algorithms;
• CA certificates that fail to decode when loading cacerts are skipped with a log entry;
• TLS/ASN1 crash connecting to some sites #8058 / PR Handle asn1 decode errors #8256 added equivalent error handling when decoding the peer Certificate message — but decode_cert_auths/2 was not covered by that fix and remains unguarded on maint-28, maint and master today.

Wrapping the public_key:pkix_normalize_name/1 call in decode_cert_auths/2 in a try/catch and skipping entries that fail to decode looks like a small, contained change, mirroring the existing patterns.

Affected versions

Reproduced on OTP 28 (erts-16.4.0.3, ssl 11.6.0.3, public_key 1.20.3.3, asn1 5.4.3), macOS aarch64. The unguarded code is identical on maint-28, maint and master as of 2026-07-05, so presumably all supported releases are affected.

Additional context

Related issues: #8058 (same crash class in the server Certificate path, fixed by #8256) and #10605 (open enhancement requesting lenient DN decoding in public_key:pkix_decode_cert/2).