Fix dependency cycles, cost budgets, scaling backtesting, prefetch limits - #459
Merged
maugauwi-hash merged 1 commit intoAug 31, 2026
Conversation
…mits Addresses four issues in the observability/performance modules: - dependency_map: reject dependency edges that would introduce a cycle (self-loop or transitive path back to the source), since a cyclic graph breaks traversal-based tooling like impacted_by/blast-radius analysis. - cost_tracking: add per-category budget thresholds (by operation or by tag key/value, covering per-vault gas and per-tenant API cost budgets), a budget-breach check run on every recorded entry, and conversions from a breach into an incidents::CreateIncidentRequest / oncall::TriggerEscalationRequest for wiring into those workflows. - predictive_scaling: add ForecastModel::backtest (and PredictiveScaler::backtest) to replay historical traffic samples and compute MAE/MAPE/RMSE, validating forecast accuracy before it drives live scaling decisions. - cache_warming: cap concurrent prefetch execution (max_concurrent_prefetches, default 10) so a large candidate set can't spike load on the origin store at once, while still prioritizing the highest-confidence targets; add a prefetch hit-rate-vs-cost metric. cache_warming, dependency_map, incidents and oncall were not wired into the crate's module tree (missing `pub mod` in lib.rs), so add them there to make the cost_tracking wiring above and their own logic reachable. Closes ethos-protocol#378 Closes ethos-protocol#379 Closes ethos-protocol#380 Closes ethos-protocol#381
|
@Realericky Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses four independent issues across the observability/performance modules:
backend/src/dependency_map.rs—POST /dependencies/discovernow rejects an edge that would introduce a cycle (a direct self-loop, or a path that already exists fromtoback tofrom), via newDependencyGraph::has_path/would_create_cyclehelpers. Rejected with422and a clear error message. The existing DOT export (GET /dependencies/graph) already serves as the graph's visualization output, documented accordingly.backend/src/cost_tracking.rs— adds configurable per-categoryBudgetThresholds, scoped either byoperationor by a tag key/value pair (covering per-vault gas and per-tenant API cost budgets).POST /admin/cost/entriesre-evaluates thresholds on every recorded entry and logs a warning on breach. NewPOST /admin/cost/budget-thresholdsandGET /admin/cost/budget-breachesendpoints, plusbreach_to_incident_request/breach_to_escalation_requesthelpers that convert a breach into anincidents::CreateIncidentRequest/oncall::TriggerEscalationRequest, wiring budget alerts into those workflows.backend/src/predictive_scaling.rs— addsForecastModel::backtest(and aPredictiveScaler::backtestconvenience wrapper) that replays historicalTrafficSampledata, forecasting ahead at each point with sufficient history and comparing against the actual sample that later arrived, returning MAE/MAPE/RMSE (BacktestResult) so model accuracy can be validated before it drives live scaling decisions.backend/src/cache_warming.rs—CacheWarmernow caps concurrent prefetch execution (max_concurrent_prefetches, default 10, configurable viaCacheWarmer::with_concurrency_limit), executingpredict_prefetch_targetsin capped chunks viafutures::future::join_allso a large candidate set can't spike load on the origin store at once — the existing confidence-descending sort means the highest-priority targets always get through first when capped. Also adds a prefetch hit-rate-vs-cost metric (WarmerStats::prefetch_hit_rate, exported viaCacheWarmer::render_prometheus).Additionally,
cache_warming,dependency_map,incidentsandoncallwere not declared inbackend/src/lib.rs's module tree at all (nopub modentry), so none of their code — including the fixes above — was actually part of the compiled crate. Added the missingpub moddeclarations so these modules (and the cost_tracking → incidents/oncall wiring) are reachable.Each corresponding doc (
docs/dependency-mapping.md,docs/cost-tracking.md,docs/predictive-scaling.md,docs/cache-warming.md) was updated to describe the new behavior.Test plan
cargo build -p ethos-protocol-backend/cargo test -p ethos-protocol-backend(not run in this environment — no Rust toolchain available)POST /dependencies/discoverwith an edge that closes a cycle and confirm422GET /admin/cost/budget-breachesreports it once crossedPredictiveScaler::backtest()/ForecastModel::backtest()against recorded traffic historyCacheWarmer::warm_cacheagainst a candidate set larger than the concurrency cap and confirm chunked executionCloses #378
Closes #379
Closes #380
Closes #381