ACA Sandboxes (preview): intermittent egress-proxy CA mismatch under Full inspection (stale/rotated CA on long-lived/stop-resumed sandboxes; fresh sandboxes validate)

[freezscholte]

Summary

With ACA dynamic sandboxes (Microsoft.App/sandboxGroups, preview) using trafficInspection: Full, the egress proxy performs TLS interception and presents a leaf signed by CN=ADC Egress Proxy Root CA. The CA delivered into the sandbox for clients to trust — /etc/ssl/certs/adc-egress-proxy-ca.crt (also /usr/local/share/ca-certificates/adc-proxy-ca.crt, and referenced by NODE_EXTRA_CA_CERTS) — has the same Subject as the leaf's issuer but a different key, so it does not validate the intercepted traffic. As a result, no client can verify TLS through the proxy, and Transform rules (header injection / credential injection), which require Full inspection, can't be used with certificate verification enabled.

Environment

• aca CLI 1.0.0-preview.1
• Microsoft.App/sandboxGroups, region North Europe
• Reproduced on a custom Ubuntu 24.04 disk and consistent with MS public disks (claude, ubuntu); on fresh and resumed sandboxes; across system trust (curl/openssl) and Node (NODE_EXTRA_CA_CERTS)

Repro

1. aca sandbox create --disk ubuntu
2. aca sandbox egress set --id <id> --default Deny --rule "api.anthropic.com:Allow" --traffic-inspection Full
3. In the sandbox:

   CA=/etc/ssl/certs/adc-egress-proxy-ca.crt
   echo | openssl s_client -connect api.anthropic.com:443 -servername api.anthropic.com 2>/dev/null | openssl x509 > /tmp/leaf.pem
   openssl verify -CAfile $CA /tmp/leaf.pem
   curl --cacert $CA https://api.anthropic.com/v1/models
   

Actual

• openssl verify -CAfile $CA /tmp/leaf.pem → error 20 at 0 depth lookup: unable to get local issuer certificate / verification failed
• curl --cacert $CA ... → exit 60 / http=000
• Node (fetch, with NODE_EXTRA_CA_CERTS set to the dropped CA) → UNABLE_TO_VERIFY_LEAF_SIGNATURE
• The proxy presents only the leaf (-showcerts shows 1 cert), so clients cannot bootstrap trust from the connection.

Key detail confirming the mismatch:

• leaf issuer_hash == dropped-CA subject_hash (same DN), but the dropped CA is a valid self-signed root (CA:TRUE, pathlen:0, Key Usage: Certificate Sign, CRL Sign, self-verifies OK) that simply did not sign the leaf → two different CAs with identical CN=ADC Egress Proxy Root CA.

Expected

The CA delivered to the sandbox (/etc/ssl/certs/adc-egress-proxy-ca.crt / NODE_EXTRA_CA_CERTS) should be the actual root that signs the proxy's intercepted certificates, so that openssl verify, curl, Node, etc. validate Full-inspected TLS without disabling verification.

Impact

• Credential injection via Transform (a headline sandbox capability for "coding agents") is unusable with TLS verification on, because Full inspection can't be trusted.
• The only client-side workaround today is disabling verification (curl -k, NODE_TLS_REJECT_UNAUTHORIZED=0, git -c http.sslVerify=false), which is unacceptable for production.

Ask

Deliver the correct (matching) egress-proxy signing CA into the sandbox trust store / NODE_EXTRA_CA_CERTS, or document the correct way to obtain it. Happy to provide more detail / hashes from a live repro.

[freezscholte]

Update / correction — narrowing scope (apologies for the initial over-broad framing).

After a clean same-timeframe side-by-side, this is intermittent, not a universal delivery bug, and not specific to custom/BYO disks:

• Fresh sandboxes (now) validate fine — both a public ubuntu disk and a custom (disk create --image) disk: the delivered CA (/etc/ssl/certs/adc-egress-proxy-ca.crt, fingerprint 7C:81:74:2F…) matches the live proxy: openssl verify → OK, curl --cacert → 200, and deny-default is enforced (example.com → 403). So Full inspection + credential Transform work as designed on fresh sandboxes.
• The earlier failures were on sandboxes created earlier and, in one case, stopped/resumed several times; their delivered CA (fingerprint 6E:52:F4:1C…) did not match the then-live proxy. The proxy CA appears to have rotated, leaving those sandboxes with a stale delivered CA (same CN=ADC Egress Proxy Root CA, different key).

So the likely root cause is CA rotation/staleness on longer-lived or stop/resumed sandboxes (the delivered adc-egress-proxy-ca.crt / NODE_EXTRA_CA_CERTS isn't refreshed when the proxy CA rotates), rather than a wrong CA being delivered universally.

Questions for the team:

1. Is the egress-proxy signing CA expected to rotate during a sandbox's lifetime (incl. across stop/resume)?
2. If so, should the in-sandbox CA file / NODE_EXTRA_CA_CERTS be refreshed on resume so long-lived sandboxes keep validating Full-inspected TLS?

Repro is non-deterministic (depends on hitting a rotation window / resume); happy to capture fingerprints + timestamps if you can point me at how the per-sandbox CA is provisioned.