feat(auth): add cache-through token caching for azure_ad and gcp_adc

[szedan-rh]

Summary

Adds apis::token_cache::TokenCache, a reusable cache-through credential
cache: a request that finds the cache stale acquires a fresh value inline,
double-checked under an exclusive lock so concurrent callers racing an
empty cache trigger at most one fetch. There is no background refresh loop
and no server-side retry/backoff; a failed fetch is simply not cached, so
the next request tries again.

• Retrofits azure_ad onto it, removing its per-instance background
  refresher thread and private tokio runtime entirely.
• Implements gcp_adc's metadata-server token fetch (GCE/GKE Workload
  Identity) on the same cache, unblocking what was previously a permanent
  503 stub. Service-account key file fetch remains a documented follow-up
  (needs JWT signing, not currently a workspace dependency) — it now fails
  closed with a clear "not implemented" reason instead of silently
  503ing forever.
• refresh_ratio is removed from both filters' config: it's meaningless
  under cache-through, since there's no scheduled ahead-of-time refresh.
• gcp_adc gains a metadata_host config field, mirroring azure_ad's
  authority_host, so a non-default or test metadata endpoint can be
  configured directly.

This is the smaller, "make it work" cache-through implementation the
team converged on as a faster path than the full refresh-ahead
credential-store design — see discussion on #811.

Related issue

Closes #811

(No equivalent tracking issue exists yet for the gcp_adc half of this
PR — that spike, praxis#554, lives in the praxis-proxy/praxis repo.)

Validation

• Unit tests — cargo test -p praxis-ai-apis and
  cargo test -p praxis-ai-filters --features azure-ad-filter,gcp-adc-filter
• Integration or functional tests — azure_ad/gcp_adc example-config
  suites in tests/integration
• make lint (workspace + experimental features)
• make test-unit

Checklist

• I reviewed every changed line and can explain the change.
• New capabilities include an example config and functional example test.
• User-facing behavior and generated documentation are updated.
• Performance-sensitive changes include appropriate benchmark or load-test evidence.
• Commits are signed and include a Signed-off-by trailer.

Breaking changes

refresh_ratio is removed from the azure_ad and gcp_adc filter
config schemas. Any deployment setting it will now fail config
validation (deny_unknown_fields) and must remove the field.

[alexsnaps]

Generally fine with the approach. There is the pathological timeouts on network partitions: probably should file an issue about that in order to follow up.

[alexsnaps]

There are some failures in CI too

[aslakknutsen]

(ai): The PR checklist claims user-facing and generated documentation were updated, but docs/filters/azure_ad.md still shows refresh_ratio in the example and docs/filters/gcp_adc.md still states token acquisition is not implemented and documents refresh_ratio. cargo xtask lint-filter-docs reports both files stale; make lint includes this check and will fail until cargo xtask generate-filter-docs is run."
(ai): "tests/integration/tests/suite/examples/azure_ad.rs lines 47-48 say authority_host is pointed at a closed port so the background refresher can never acquire a token. The refresher was removed; the comment should say inline cache-through fetch instead to avoid misleading future readers."

[aslakknutsen]

The approach look good. I'm wondering if we could iterate on it and possible split it up into two separate times;
soft expiry (ttl-margin) and hard expiry (ttl).

• At "soft expiry" we would trigger a refresh, blocking the single request. The rest of the requests could get the "old still valid token" while it's being refreshed.
• If we hit the "hard expiry" we would block requests waiting for the refresh

And I think we need to address what happens if the the refresh never happens, e.g. the server is down. Not sure we can just "queue" up requests for ever. At some point we need to start returning some form of error to the client to free up the requests.

[szedan-rh]

The approach look good. I'm wondering if we could iterate on it and possible split it up into two separate times; soft expiry (ttl-margin) and hard expiry (ttl).

• At "soft expiry" we would trigger a refresh, blocking the single request. The rest of the requests could get the "old still valid token" while it's being refreshed.
• If we hit the "hard expiry" we would block requests waiting for the refresh

And I think we need to address what happens if the the refresh never happens, e.g. the server is down. Not sure we can just "queue" up requests for ever. At some point we need to start returning some form of error to the client to free up the requests.

I think soft expiry (ttl-margin) / hard expiry (ttl) is a reasonable idea to explore. On the failure scenario fair pushback, I was overstating it as "IdP down," and you're right that's rare for GCP/Azure specifically.

The sharper risk isn't IdP availability though, it's request duration: even with a perfectly healthy IdP, a pass-through request that picks up the token right as it crosses into the soft -> hard window could still be mid-flight when hard expiry hits, purely because its own request ran longer than the margin nothing to do with refresh succeeding or failing. That's the actual reason the margin exists today (never hand out a token that could outlive the request holding it), so pass-through would need an explicit answer for that, not just a faster/rarer refresh-failure path.

I'd rather not fold this into the current PR it adds real synchronization complexity (a second "who's refreshing" signal separate from the read/write lock) on top of what we deliberately scoped down to. If we want to pursue it, I'd want to spec it as its own follow-up with an explicit answer to "is it OK for a request to receive a token that might expire before its own response completes," rather than assume the hard-expiry backstop dissolves that risk.

[szedan-rh]

@alexsnaps / @aslakknutsen can you please take another look?

[aslakknutsen]

The sharper risk isn't IdP availability though, it's request duration: even with a perfectly healthy IdP, a pass-through request that picks up the token right as it crosses into the soft -> hard window could still be mid-flight when hard expiry hits, purely because its own request ran longer than the margin nothing to do with refresh succeeding or failing. That's the actual reason the margin exists today (never hand out a token that could outlive the request holding it), so pass-through would need an explicit answer for that, not just a faster/rarer refresh-failure path.

The refresh tokens doesn't have TTL for seconds or minutes right, we're talking closer to 1h+?

While not perfect, we could make the margin for soft expiry something like 5m and also use a margin for hard expiry at e.g. 1m. It creates over time a few more refresh calls, but also gives plenty of time to try to refresh even with a few retries and even pre hard expiry requests have a certain time buffer to complete before it's invalid.

[szedan-rh]

The sharper risk isn't IdP availability though, it's request duration: even with a perfectly healthy IdP, a pass-through request that picks up the token right as it crosses into the soft -> hard window could still be mid-flight when hard expiry hits, purely because its own request ran longer than the margin nothing to do with refresh succeeding or failing. That's the actual reason the margin exists today (never hand out a token that could outlive the request holding it), so pass-through would need an explicit answer for that, not just a faster/rarer refresh-failure path.

The refresh tokens doesn't have TTL for seconds or minutes right, we're talking closer to 1h+?

While not perfect, we could make the margin for soft expiry something like 5m and also use a margin for hard expiry at e.g. 1m. It creates over time a few more refresh calls, but also gives plenty of time to try to refresh even with a few retries and even pre hard expiry requests have a certain time buffer to complete before it's invalid.

Fair point on the TTL scale with hour-long tokens, your numbers hold up: pass-through requests would still have ≥1m of guaranteed validity, more conservative than the 30s margin we already ship today.
I'd keep this out of the current PR though worth its own follow-up with configurable margins (request durations vary a lot across our use cases, so I wouldn't hardcode 5m/1m). I'll fold it into ai#864 so it doesn't get lost.