Skip to content

The maintenance run ledger records a row per sweep, so its volume scales with catalogs × tasks × pods × frequency #131

Description

@jghoman

Six catalogs on dev today, and the recent-runs table is entirely rows saying nothing happened:

cleanup   removed 0, absent 0
expiry    expired 0 snapshots, queued 0 files, floor 0
cleanup   removed 0, absent 0
...
20 quiet runs hidden   |   No runs match these filters.

MaintenanceRunStore.record writes one row per (catalog, task, sweep), unconditionally. So ledger volume is catalogs × tasks × pods × frequency — and every one of those factors grows in the direction that makes it worse. Six catalogs is a developer's laptop-scale number; a real fleet is hundreds. At 500 catalogs, even an hourly expiry sweep writes 500 rows an hour to say that nothing expired.

Pacing the loops (#122) divides the constant. It does not change the shape: the ledger still grows with catalog count for work that never happened.

Proposal

Record a run only when there is something to record — it changed something, or it failed. Keep liveness as a per-(catalog, task) last_swept_at timestamp, updated in place, which is what the status header actually needs to answer "is this loop alive?".

That makes the runs table a log of events rather than a heartbeat, at any catalog count. A screenful would then be things that happened, which is the state the filters in #120 are approximating client-side.

Why this is more than cosmetics

  • It removes the need for the filters to exist. webui: let the runs table hide the runs that did nothing #120 hides quiet runs client-side and webui: show tables, rows and size per catalog #121 fixes the paging bug that creates (filtering a server page client-side means hidden rows are never replaced, so an interesting row can be evicted from the window and become unreachable). If quiet runs were never written, there would be nothing to filter and no window to evict from.
  • Retention exists but is the wrong tool. MaintenanceRunStore.deleteOlderThan prunes by age from the cleanup sweep, so the ledger is bounded — but bounded by time, not by usefulness. At high catalog counts the window fills with no-ops and the genuinely interesting rows age out just as fast as the noise.
  • Write cost. Every sweep is currently an INSERT against a shared table on the catalog's own database, whether or not the sweep did anything.

Design questions worth settling in the issue

  • Where does last_swept_at live — a column on an existing per-catalog row, or a small table keyed by (catalog, task)? It is written every sweep, so it should not contend with anything hot.
  • Does a manual trigger always record, even when it changed nothing? I would say yes: someone asked, and the answer ("nothing to do") is the information.
  • Does "changed something" reuse the predicate the webui now has (isQuietRun in webui/src/components/maintenance.tsx)? If the server gains its own version there are two implementations of a subtle rule — the webui's enumerated delete_files_queued, which the renderer never displays, and a fifth task (verify) that an earlier summary of the rule omitted entirely. Whatever lands should make divergence impossible or loudly detectable.
  • What happens to the existing rows and to webui: let the runs table hide the runs that did nothing #120's filters once this is in? The filters become harmless but redundant; worth deciding whether they stay.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions