Chaos scenarios, retry jitter validation, timeout convergence tests, consensus reconciliation

[Haroldwonder]

Summary

Four reliability/testing fixes, each independently scoped:

• Closes Add Chaos Testing Scenario for Cascading Failures #370 — Combined-failure chaos scenarios. chaos.rs gains two scenarios: (1) cache + one replica down simultaneously, asserting fallback::cascade routes around both to a healthy target instead of erroring, with a heal/recovery check; (2) circuit breaker + bulkhead interacting under 20-way concurrent load, asserting no panics/deadlocks and that the breaker fast-rejects calls once tripped even after they've passed the bulkhead. This wires the previously-unused fallback.rs and bulkhead.rs modules into the crate, which surfaced a real bug in Bulkhead::acquire: it gated every request against max_queue_size, even ones with an immediately-free concurrency slot, so max_queue_size: 0 rejected all traffic outright regardless of max_concurrent. Fixed as part of this PR (see bulkhead.rs diff) since it's exactly the kind of latent bug this issue exists to catch. Documented in docs/chaos-testing.md, including honest gaps (both scenarios drive the modules directly rather than through the live HTTP router).

• Closes Implement Retry Policy Jitter Bounds Validation #371 — Retry policy jitter/backoff validation. RetryPolicy::validate() checks max_attempts > 0, base_delay_ms <= max_delay_ms, and that multiplier is finite and positive (rejecting zero/negative/NaN/infinite, which would otherwise produce degenerate or blown-up backoff delays). Wired into the create_retry_policy endpoint. New tests cover boundary-valid and invalid configurations, plus delay-bounds tests for all three jitter modes (None/Full/Equal — Equal previously had no boundedness test at all).

• Closes Add Timeout Adaptation Convergence Test #372 — Timeout adaptation convergence tests. New tests confirm current_timeout converges to the steady-state latency within exactly min_samples iterations and then holds stable (no oscillation), the predictive EMA settles near the steady-state value, and a latency spike recovers to baseline once it ages out of the fixed-size rolling window (after window_size further samples). Documented in a new "Convergence Characteristics" section in docs/timeout-adaptation.md.

• Closes Implement Consistency Verification Scheduled Reconciliation #373 — Scheduled consensus reconciliation. scheduler.rs now runs NodeCache::check_and_resolve() every 5 minutes, publishes the result as /metrics counters/gauge (ethos_protocol_consensus_checks_total, _conflicts_total, _consistent), and opens a Sev3 incident via incidents.rs when conflicts are found. This required wiring the previously-orphaned incidents.rs module into the crate (pub mod incidents; was missing) and mounting its HTTP API for the first time. Documented in docs/consistency-verification.md.

Test plan

• cargo check --lib --bin ethos-protocol-backend — clean (only pre-existing, unrelated warnings)
• cargo test --lib — 393 passed, 0 failed, including all new coverage and the fixed bulkhead::tests

[drips-wave]

@Haroldwonder Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits