The console shows live bytes only, so nothing says how much object storage a catalog is actually using

[jghoman]

The instance header reads 151 GiB · 1.7B rows. That is live data — CatalogMetrics.kt:250 sums file_size_bytes under f.end_snapshot IS NULL, so a file drops out of the number the moment compaction supersedes it.

But the object is still in the bucket. It stays readable at older snapshots until expiry marks it and cleanup drains it, and on a catalog with retention off (every catalog on dev tonight) expiry never marks it at all — which is why the cleanup sweeps have reported 0 queued all evening while compaction rewrote gigabytes.

So today the console can tell you how much data a catalog holds, and nothing tells you how much S3 it is using. After an evening of compaction those two numbers are meaningfully apart, and the second one is the one with a bill attached.

What to show

Two counts, named so they cannot be confused:

• Live — what the header shows now: files with end_snapshot IS NULL. The data a reader sees at head.
• Total — every registered file still present in object storage, superseded ones included: live, plus files retained for time travel, plus files queued for removal but not yet drained.

The gap between them is the interesting number on its own: it is time-travel retention plus compaction debt awaiting cleanup, and watching it grow is how you notice that expiry is disabled or that cleanup is not draining.

Worth settling while designing it

• Where the boundary sits. A superseded file is "retained for time travel" until expiry floors past it, then "queued for removal" in hog_file_removal, then gone. Those are three different states with three different remedies (retention policy, expiry cadence, cleanup cadence), so a single "total" may be less useful than live / retained / queued.
• Orphans are invisible to all of it. Files uploaded by a failed commit or a failed compaction are in the bucket and in no catalog row (see Compaction buffers whole objects in memory, so the largest single file must fit twice #137 and the orphan counters in Size the JVM heap, and budget compaction groups on rows rather than bytes #133). A number derived from hog_data_file cannot see them by construction — which should be said plainly wherever the total appears, or someone will reconcile it against the S3 console and find it short.
• DV masking. The row count is already documented as "gross of DV masking", so deleted rows still count. If a "live rows" figure is going to sit next to a "total", it is worth deciding whether either should net out deletions.
• Cost. live_bytes is a scalar subselect on an indexed predicate. A total over all non-drained files is a different scan, and the sampler exists precisely so the manifest is not summed per request — so this belongs in the sampler's pass, not in the header's query path.

Related

#133, #137 (orphaned objects nothing counts), #131 (the sampler and ledger scaling), and the expiry/retention settings that decide how long the gap stays open.