Skip to content

Health-routing hysteresis, backup checksum verification, DR automation hooks, slice inheritance entry points - #460

Merged
maugauwi-hash merged 2 commits into
ethos-protocol:mainfrom
richardiyamura:fix/dr-health-backup-inheritance-374-375-376-37
Aug 31, 2026
Merged

Health-routing hysteresis, backup checksum verification, DR automation hooks, slice inheritance entry points#460
maugauwi-hash merged 2 commits into
ethos-protocol:mainfrom
richardiyamura:fix/dr-health-backup-inheritance-374-375-376-37

Conversation

@richardiyamura

@richardiyamura richardiyamura commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Addresses four issues:

#374 — Health-Based Routing Flapping Prevention
health_routing.rs marked an endpoint healthy again as soon as it recorded a single success, since is_healthy() was derived directly from consecutive_failures (which resets to 0 on any success). That let a backend flap in and out of rotation. Added a sticky healthy flag with two separate thresholds: UNHEALTHY_THRESHOLD (5 consecutive failures) marks it unhealthy, RECOVERY_THRESHOLD (3 consecutive successes) marks it healthy again — a real hysteresis band instead of a single threshold read both ways. Documented in docs/health-based-routing.md.

#375 — Backup Validation Checksum Verification
backup_validation.rs only checked file presence/magic-bytes and a simulated restore, not whether the bytes had silently changed since backup creation. Added BackupValidator::compute_checksum (SHA-256), a BackupChecksumRecord to pair a backup with the checksum recorded at creation time, and validate_backup_with_checksum, which fails validation and opens a Sev2 incident via the (previously unregistered) incidents module on a mismatch.

#376 — Disaster Recovery Runbook Automation Hooks
Added a new dr_automation module exposing the runbook's highest-risk manual steps (pause, unpause, restore_vault) as admin endpoints, gated by a two-phase confirmation-token flow (POST /admin/dr/actions → token, POST /admin/dr/actions/:token/confirm → executes, 5-minute expiry) so a destructive action can't fire from a single accidental request. Each confirmed action shells out to the same stellar contract invoke commands already documented in the runbook. docs/disaster-recovery-runbook.md updated to reference the new API alongside the manual steps.

#377 — Saga Compensation Idempotency Guarantee
saga.rs's CompensationRegistry had no memory of which compensations had already run,
so retrying a saga after a crash (or an external system retrying the same logical
operation) could re-invoke a compensation that already succeeded — e.g. double-refunding
or double-releasing a resource. CompensationRegistry now tracks, per step name,
whether that step's compensation already completed successfully; a repeat call is a
no-op instead of re-invoking the function. SagaStepRecord.compensation_already_ran
records when that guard fires. A failed compensation is deliberately left retryable.
Documented in docs/saga-pattern.md.

  • No new tests were added and the build was not run, per instructions for this PR.
  • incidents.rs existed on disk but wasn't declared as a module anywhere in the crate; it's now wired in via pub mod incidents; in backend/src/lib.rs since backup_validation's checksum-mismatch alerting depends on it.
  • dr_automation follows the same standalone-module pattern already used by health_routing/incidents (its own Axum state, not yet wired into main.rs::build_router), consistent with how those modules currently sit in this codebase.

Closes #374
Closes #375
Closes #376
Closes #377

…ation hooks, and slice inheritance contract entry points

- Health-based routing (ethos-protocol#374): separate mark-unhealthy / mark-healthy-again
  thresholds (UNHEALTHY_THRESHOLD / RECOVERY_THRESHOLD) with a sticky
  `healthy` flag so a single success after crossing the unhealthy threshold
  no longer flips an endpoint back into rotation immediately; documented in
  docs/health-based-routing.md.
- Backup validation (ethos-protocol#375): SHA-256 checksum computation, a
  BackupChecksumRecord to pair a backup with the checksum recorded at
  creation time, and validate_backup_with_checksum which fails validation
  and opens a Sev2 incident via incidents::open_incident on mismatch.
  Wires the previously-unregistered `incidents` module into the crate.
- DR runbook automation (ethos-protocol#376): new dr_automation module exposing the
  pause/unpause/restore_vault manual runbook steps as admin endpoints
  behind a two-phase confirmation-token flow; runbook updated to document
  the new API alongside the manual CLI steps.
- Slice inheritance (ethos-protocol#37): exposes the existing template_inheritance
  module (create/inherit/resolve/cycle-detection, already implemented at
  the module level) as public contract entry points on TtlVaultContract,
  which weren't previously reachable through the contract's public API.

Closes ethos-protocol#374
Closes ethos-protocol#375
Closes ethos-protocol#376
Closes ethos-protocol#37
@drips-wave

drips-wave Bot commented Aug 30, 2026

Copy link
Copy Markdown

@richardiyamura Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

The fourth issue for this PR was actually ethos-protocol#377 (Saga Compensation
Idempotency), not ethos-protocol#37 (Slice Inheritance Chain) — reverting the
template_inheritance contract entry points added for ethos-protocol#37 and replacing
them with the ethos-protocol#377 fix.

Saga compensation idempotency (ethos-protocol#377): CompensationRegistry now tracks,
per step name, whether that step's compensation already succeeded. A
retried saga run (e.g. after a crash, or an external retry re-triggering
the same logical operation) skips re-invoking a compensation that already
completed successfully instead of risking a double-refund or
double-release; SagaStepRecord.compensation_already_ran records when that
guard fires. A compensation that failed is intentionally left retryable.
Documented in docs/saga-pattern.md.

Closes ethos-protocol#377

Copy link
Copy Markdown
Contributor

Merged to main via conflict resolution. All required changes integrated.

@maugauwi-hash maugauwi-hash reopened this Aug 31, 2026
@maugauwi-hash
maugauwi-hash merged commit 8416c3e into ethos-protocol:main Aug 31, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants