Operator run index scale follow-ups

[marcusrbrown]

Summary

Track non-blocking follow-ups from the GET /operator/runs implementation merged in #1035.

Follow-ups

• Evaluate run-index read amplification: the endpoint can perform up to authorized repos × MAX_RUNS_PER_REPO run-state object reads per request, while returning at most 100 summaries. Current rate limits bound blast radius, but higher repo/run counts may need an early global short-circuit, smaller per-repo cap, or a stronger index.
• Consider deduplicating bindings before the authz fan-out cap: duplicate channel bindings for one owner/repo can consume cap slots before owner/repo dedup, reducing unique repo coverage below the nominal cap.
• Define the long-term run-state retention policy: listWithMetadata sorts after bounded S3 pagination, so extremely large per-repo run-state prefixes can miss newer objects beyond the list iteration cap. S3 lifecycle cleanup is the likely operational mitigation.

Context

These are not correctness or security blockers for #1035. They are scale/operational fairness follow-ups for larger installations or long-lived deployments.

[fro-bot]

Triage: Operator run index scale follow-ups

Summary

This is a tracking issue for three non-blocking scale/operational-fairness follow-ups from the GET /operator/runs run-index route merged in #1035 (plan). None are correctness or security blockers — they affect larger installations and long-lived deployments. The issue body's inline-code placeholders were dropped by markdown rendering; I've reconstructed the concrete references against the merged code below.

Reproduction

Not reproducible in CI — all three are scale/operational properties of the deployed gateway against S3 at high repo/run counts, not defects with a deterministic test trigger. Instead I verified each claim against the merged source:

1. Run-index read amplification — CONFIRMED

• runs-route.ts:204 caps the authz fan-out at MAX_REPOS_AUTHZ_FANOUT = 100.
• For each authorized repo, runIndex.listRunsForRepo() reads up to MAX_RUNS_PER_REPO = 200 run-state objects (run-index.ts:132, :499), each a sequential getObject (run-index.ts:443-458).
• Worst case: 100 repos × 200 reads = ~20,000 sequential object reads to return at most MAX_RUNS_PER_LISTING = 100 summaries (runs-route.ts:57).
• Current mitigations bounding blast radius: operator-keyed rate limit 20/min (runs-route.ts:74), the 100-repo authz cap, and the per-repo 200 cap. At higher repo/run counts these may still be insufficient — a global early short-circuit (stop once 100 newest collected), a smaller per-repo cap, or a stronger index would help.

2. Dedup-after-cap fairness — CONFIRMED

• Bindings are keyed by channelId, so one owner/repo can have multiple channel bindings.
• In runs-route.ts:200-204 the denylist filter and .slice(0, MAX_REPOS_AUTHZ_FANOUT) run before the owner/repo dedup, which happens later at runs-route.ts:235-242 (the scannedRepos set).
• Consequence: duplicate bindings for the same repo consume authz-cap slots, so the number of unique repos covered can fall below the nominal 100. Deduping by owner/repo before the cap would restore full unique-repo coverage.

3. Run-state retention policy — CONFIRMED

• run-index.ts:497-499 sorts by lastModified desc and takes the newest MAX_RUNS_PER_REPO, but the sort operates only on what listWithMetadata(prefix) already returned.
• If a single repo's run-state prefix grows beyond the S3 list-iteration bound, newer objects past that bound can be missed — "newest-first" becomes only locally true (the plan's "Timeout/newest-first caveat" already documents this honestly).
• The plan already recommends the operational mitigation: an S3 lifecycle rule deleting terminal run-state objects after N days (plan "Documentation / Operational Notes" + "Risks & Dependencies"). There is no application-level pruning today.

Proposed next steps

Order by effort vs. impact; each is independently shippable and non-blocking:

1. Dedup bindings before the authz cap (smallest, highest fairness win). Move the owner/repo dedup ahead of runs-route.ts:204's .slice() so the cap counts unique repos, not bindings. Low risk, pure ordering change, easy to unit-test (binding list with duplicate repos → unique-repo coverage == cap). Good first PR.
2. Add a global newest-N short-circuit to bound read amplification. Track collected summaries across the per-repo loop and stop scanning once MAX_RUNS_PER_LISTING newest are gathered, or lower MAX_RUNS_PER_REPO. Requires care to preserve true newest-first ordering across repos (may need a per-repo newest-K merge rather than naive early-exit). Medium effort; pairs naturally with feat: update repo settings for agent #1.
3. Document + provision the S3 lifecycle retention policy. Zero-code operational mitigation — add the lifecycle rule to deploy/ docs and recommend it for long-lived deployments. The per-repo cap already bounds per-request cost; retention bounds prefix growth so the newest-first guarantee holds. Application-level pruning is a larger, separate effort and not needed if lifecycle cleanup is in place.

Recommendation: Land #1 first (trivial, removes a real fairness gap), then #3 (doc/ops, unblocks long-lived installs), then #2 when a higher-binding-count deployment actually materializes — consistent with the plan's "defer the route timeout until a high-binding deployment needs one" stance. No action is required for the current single-operator deployment.

---

Run Summary

Field	Value
Event	issues
Repository	fro-bot/agent
Run ID	28217181879
Cache	hit
Session	ses_0fdcef909fferlPyiqGzdQHZ7J