Add Chaos Testing Scenario for Cascading Failures

[maugauwi-hash]

Labels: testing, reliability
Priority: Medium
Estimated Time: 2.5 hours

Description:
backend/src/chaos.rs supports fault injection, but docs/chaos-testing.md should be checked for a scenario that chains multiple simultaneous failures (e.g., cache down + one replica down) to verify the system degrades rather than cascades into full outage.

Tasks:

• Add a chaos scenario combining cache unavailability with replica failure
• Assert the system falls back to fallback.rs behavior instead of erroring
• Add a chaos scenario for circuit breaker + bulkhead interacting under load
• Document findings and any gaps in docs/chaos-testing.md

[Bezzy2105]

@Bezzy2105 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi maintainer, I can work on this issue please assign to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Bezzy2105 to this issue.

[Haroldwonder]

@Haroldwonder has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

I'd like to work on this, please

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Haroldwonder to this issue.

[drips-wave]

Congratulations, @Haroldwonder! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on August 31, 2026.

🧑‍💻 @Haroldwonder: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox!

@Haroldwonder's PR #449 was approved and merged by @maugauwi-hash.

🏆 @Haroldwonder: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (812 total points). Track your full progress on GrantFox.

👏 Great work, @Haroldwonder! Keep contributing to ethos-protocol.