Skip to content

feat(canary): add bounded fleet tripwires (#152) - #166

Draft
tcballard wants to merge 2 commits into
codex/enterprise-shadow-evidencefrom
codex/enterprise-canary-tripwires
Draft

feat(canary): add bounded fleet tripwires (#152)#166
tcballard wants to merge 2 commits into
codex/enterprise-shadow-evidencefrom
codex/enterprise-canary-tripwires

Conversation

@tcballard

Copy link
Copy Markdown
Contributor

Summary

Adds bounded production canary rollout and fleet-wide quality tripwires for #152.

Scope

Included

  • deterministic request/workspace/key/approved-cohort assignment
  • named candidate-route canary gate after the authoritative scored decision
  • Redis fixed-window exposure ceiling and fail-closed state requirement
  • idempotent, prompt-free fleet counters for requests, errors, latency, cost, and quality observations
  • automatic error-rate, latency, cost-multiplier, and quality-loss tripwires
  • shared rollback hold state and operator audit record
  • operator-authenticated wf-canary-v1 status endpoint
  • bounded config parsing/rendering, tests, ADR, and operator documentation

Excluded

Product / Architecture Decisions

  • Canary assignment is a post-decision gate; privacy, capability, access, and deployment eligibility are rechecked before exposure.
  • Redis is required when a rollout is enabled so no replica can expand exposure independently.
  • Missing identity, candidate eligibility, telemetry, or fleet state suppresses canary traffic and leaves the normal route active.
  • Tripwires evaluate only after the configured minimum unique observations; missing cost/quality samples are not imputed.
  • Rollback state is shared and bounded; the deterministic route remains the sole scoring authority.

User-Facing Contract

CLI / Human Output

No CLI command changes. Operators can inspect GET /v1/canary (or /router/canary) for versioned rollout state and tripwire counters.

JSON Output

Canary responses carry x-wayfinder-router-canary with the rollout ID when admitted. wf-canary-v1 reports bounded counters and the shared rollback reason without prompt or response content.

Exit Codes

No CLI exit-code changes.

Verification

Ran

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo test --workspace --all-targets --locked (with local socket access)

Covered

  • deterministic assignment and scope handling
  • sparse samples and each tripwire family
  • bounded/idempotent MemoryStateBackend counters and rollback state
  • Redis-backed script compilation and typed state mapping
  • config bounds and round-trip rendering
  • status endpoint and existing Chat/streaming/concurrency behavior

Review Path

Stacked on #165 (evidence reports), targeting codex/enterprise-shadow-evidence. Review after #165.

Notes For Reviewer

This PR intentionally requires Redis for enabled canaries. A degraded or unavailable state backend cannot increase exposure; it suppresses new assignments and leaves the incumbent route in control.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant