Partition drop: end every file in a partition range in one snapshot, with a per-table retention policy

[jghoman]

events_raw (the millpond→hoglake table, partitioned day(timestamp)) is append-only: no UPDATE, no row DELETE. Its only retention operation is "drop everything older than N days". hoglake has no way to say that today. The options are row deletes through deletion vectors (a DV per file for a whole-file delete, then compaction to reclaim), TRUNCATE (the whole table), or snapshot expiry (time travel, which this table does not need).

What is needed

A partition-level drop: end every live data file (and its DV) whose partition values satisfy a predicate, in one snapshot, without touching any object. Two entry points:

1. Explicit: POST /catalogs/{c}/namespaces/{ns}/tables/{t}/partitions/drop with a predicate on the table's partition fields (day(timestamp) < 2026-09-16, or a list of partition tuples), the usual expected_table_uuid / read_snapshot guards, and an idempotency key so a lost response is not a double drop. Returns the snapshot id and the count of files ended.
2. Policy: per-table partition_retention ({ field, keep: 7d }) evaluated by the expiry sweep, so a table drops its own tail without a client.

Mechanism

• hog_file_partition_value already carries each file's partition tuple, so the predicate is a metadata query; ending files is FileRepo.endLiveFiles for the matching ids plus the matching hog_delete_file rows, at a freshly allocated snapshot under the catalog commit lock. Row-id ranges are never reused, so nothing else changes.
• Publish table_deleted_from for the table so guarded writers with an older read_snapshot conflict, exactly as a DV publication does; the changefeed then reports the ended files as deletes, which is what a downstream consumer must apply.
• Objects go through the existing pipeline: once the retention floor passes the drop snapshot, expiry queues the ended files and cleanup deletes them. No new deletion path, no new ledger reason.
• Compaction must not rewrite a partition that policy is about to drop. The planner reads the table's partition_retention and excludes partitions inside the drop window; rewriting them is pure amplification (MaintenanceSummarySampler should exclude them from debt too).
• VerifyService gets one invariant: no live file with a partition value inside the retention window.

Why not DVs

A DV-per-file delete writes N puffin objects, a commit that registers all of them, and then a compaction rewrite that materializes the deletes, for rows nobody will read again. Ending the file rows is one UPDATE.

Retention for append-only tables

For a table like this, snapshot_retention_seconds only needs to cover the longest in-flight read (a scan planned at S still reads files compaction retired after S), and consumer_floor protects changefeed consumers independently. An hour is generous.

Related: #157 (TRUNCATE is the whole-table special case of this), #160 (DV publication, the row-level alternative), #140 (retention as a curve), the millpond events_raw dev instance (charts #16117).

[jghoman]

Refinement from discussion: the explicit endpoint takes an enumerated list of partition tuples and drops all of them in one atomic commit (one snapshot, one table_deleted_from, one receipt under the idempotency key). No wildcards or predicates on the wire: team_id=42/day=* is not accepted; the caller enumerates every (team_id, day) tuple it means. The list is the request payload the receipt fingerprints, so a retry drops exactly the same set or replays the original result. A tuple that matches no live file is a no-op inside the commit, not an error, so a retry after a partial read of the response is still idempotent; the response reports files ended per tuple. Range predicates stay with the policy side (partition_retention), where the server derives the tuples itself.