Health-routing hysteresis, backup checksum verification, DR automation hooks, slice inheritance entry points - #460
Merged
maugauwi-hash merged 2 commits intoAug 31, 2026
Conversation
…ation hooks, and slice inheritance contract entry points - Health-based routing (ethos-protocol#374): separate mark-unhealthy / mark-healthy-again thresholds (UNHEALTHY_THRESHOLD / RECOVERY_THRESHOLD) with a sticky `healthy` flag so a single success after crossing the unhealthy threshold no longer flips an endpoint back into rotation immediately; documented in docs/health-based-routing.md. - Backup validation (ethos-protocol#375): SHA-256 checksum computation, a BackupChecksumRecord to pair a backup with the checksum recorded at creation time, and validate_backup_with_checksum which fails validation and opens a Sev2 incident via incidents::open_incident on mismatch. Wires the previously-unregistered `incidents` module into the crate. - DR runbook automation (ethos-protocol#376): new dr_automation module exposing the pause/unpause/restore_vault manual runbook steps as admin endpoints behind a two-phase confirmation-token flow; runbook updated to document the new API alongside the manual CLI steps. - Slice inheritance (ethos-protocol#37): exposes the existing template_inheritance module (create/inherit/resolve/cycle-detection, already implemented at the module level) as public contract entry points on TtlVaultContract, which weren't previously reachable through the contract's public API. Closes ethos-protocol#374 Closes ethos-protocol#375 Closes ethos-protocol#376 Closes ethos-protocol#37
|
@richardiyamura Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
The fourth issue for this PR was actually ethos-protocol#377 (Saga Compensation Idempotency), not ethos-protocol#37 (Slice Inheritance Chain) — reverting the template_inheritance contract entry points added for ethos-protocol#37 and replacing them with the ethos-protocol#377 fix. Saga compensation idempotency (ethos-protocol#377): CompensationRegistry now tracks, per step name, whether that step's compensation already succeeded. A retried saga run (e.g. after a crash, or an external retry re-triggering the same logical operation) skips re-invoking a compensation that already completed successfully instead of risking a double-refund or double-release; SagaStepRecord.compensation_already_ran records when that guard fires. A compensation that failed is intentionally left retryable. Documented in docs/saga-pattern.md. Closes ethos-protocol#377
Contributor
|
Merged to main via conflict resolution. All required changes integrated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses four issues:
#374 — Health-Based Routing Flapping Prevention
health_routing.rsmarked an endpoint healthy again as soon as it recorded a single success, sinceis_healthy()was derived directly fromconsecutive_failures(which resets to 0 on any success). That let a backend flap in and out of rotation. Added a stickyhealthyflag with two separate thresholds:UNHEALTHY_THRESHOLD(5 consecutive failures) marks it unhealthy,RECOVERY_THRESHOLD(3 consecutive successes) marks it healthy again — a real hysteresis band instead of a single threshold read both ways. Documented indocs/health-based-routing.md.#375 — Backup Validation Checksum Verification
backup_validation.rsonly checked file presence/magic-bytes and a simulated restore, not whether the bytes had silently changed since backup creation. AddedBackupValidator::compute_checksum(SHA-256), aBackupChecksumRecordto pair a backup with the checksum recorded at creation time, andvalidate_backup_with_checksum, which fails validation and opens a Sev2 incident via the (previously unregistered)incidentsmodule on a mismatch.#376 — Disaster Recovery Runbook Automation Hooks
Added a new
dr_automationmodule exposing the runbook's highest-risk manual steps (pause, unpause,restore_vault) as admin endpoints, gated by a two-phase confirmation-token flow (POST /admin/dr/actions→ token,POST /admin/dr/actions/:token/confirm→ executes, 5-minute expiry) so a destructive action can't fire from a single accidental request. Each confirmed action shells out to the samestellar contract invokecommands already documented in the runbook.docs/disaster-recovery-runbook.mdupdated to reference the new API alongside the manual steps.#377 — Saga Compensation Idempotency Guarantee
saga.rs'sCompensationRegistryhad no memory of which compensations had already run,so retrying a saga after a crash (or an external system retrying the same logical
operation) could re-invoke a compensation that already succeeded — e.g. double-refunding
or double-releasing a resource.
CompensationRegistrynow tracks, per step name,whether that step's compensation already completed successfully; a repeat call is a
no-op instead of re-invoking the function.
SagaStepRecord.compensation_already_ranrecords when that guard fires. A failed compensation is deliberately left retryable.
Documented in
docs/saga-pattern.md.incidents.rsexisted on disk but wasn't declared as a module anywhere in the crate; it's now wired in viapub mod incidents;inbackend/src/lib.rssincebackup_validation's checksum-mismatch alerting depends on it.dr_automationfollows the same standalone-module pattern already used byhealth_routing/incidents(its own Axum state, not yet wired intomain.rs::build_router), consistent with how those modules currently sit in this codebase.Closes #374
Closes #375
Closes #376
Closes #377