[User Story] Billing-grade metric export & persistence

[arasHi87]

Description

As a billing / consumption system, I want cumulative, restart-safe per-tenant and per-server usage counters exposed over Prometheus so that I can bill network traffic accurately across agent restarts, crashes, and reboots.

Acceptance Criteria

A detailed list of conditions that must be met for the story to be accepted. These statements must focus on functional behavior and constraints rather than specific code-level implementation.

• lachesis_bytes_total / lachesis_packets_total are emitted by a custom prometheus.Collector (never a CounterVec) so counters never reset and rate() never goes negative.
• A Write-Ahead Log persists GlobalState; counters are restored on boot with no double-counting.
• Direction is exported as VM-frame tx/rx; the billing model and consumption contract are documented.
• Per-server billing-grade export is available (server_id + external_network resolved), suitable for downstream CMP billing.

Story Points: ❓

Estimated at the Feature level — see child Features (#99, #107, #146, #150). User Stories are not separately pointed on the [Bigstack] scrum board (Story Points live on the pointed atom — this story's Features).

Output artifacts (Definition of Done)

Beyond code, docs, and config changes, completing this issue must also deliver:

• Handbook knowledge update — land the durable, team-readable knowledge from this
  work into the bigstack-handbook cubecos kb (kb/cubecos/…) via
  /bigstack-core:save-to-handbook (Topic / Runbook / Known-issue / ADR as fits).

Progress

• 2026-07-13 — Contract-7 monotonicity hardened (Bug [Bug] Ghost sweep re-buckets deleted VMs' historical bytes to unknown — per-tenant series non-monotonic, under-billing #111, Sprint 7). The ghost sweep was silently re-attributing deleted VMs' historical bytes to tenant_id="unknown", making per-tenant series non-monotonic and under-billing under the prescribed subtraction consumption — a direct violation of this story's "counters never reset" acceptance criterion. Fixed by the settled-bytes fold (Settle dying flows' bytes to their tenant so per-tenant series stay monotonic across VM churn #66, merged): bytes fold into a per-(tenant, zone, direction) settled accumulator when their attribution dies (VM deleted + swept, or live port reassigned), the WAL schema is now v2 (additive settled section, v1 loads clean), and the fix is live-validated on staging. One follow-up remains on the bug: the scenariotest MAC-reuse live scenario.

• 2026-07-29 — Bug [Bug] Pressure-relief eviction leaves a stale delta baseline — a busy flow's bytes go unbilled (17.6% counted, measured) #287 CLOSED: pressure-relief eviction was unbilling busy flows (Sprint 8). The same class as [Bug] Ghost sweep re-buckets deleted VMs' historical bytes to unknown — per-tenant series non-monotonic, under-billing #111 — a GC mechanism silently corrupting exposed billing counters — but by loss rather than re-attribution, so it violated this story's "counters never reset / no double-counting" criterion from the other side. Pressure-relief GC deleted a flow's telemetry_map entry without invalidating its userspace delta baseline, so the re-created kernel entry was differenced against a counter that no longer existed. Measured on dev-cmp: 17.6% of transmitted bytes counted, and worse the busier the node (eviction only reaches an active flow once map population is high enough). Fixed twice over: PR Invalidate a flow's delta baseline when pressure-relief evicts its kernel entry #289 added an out-of-band BaselineInvalidator, then ADR 0014 / PR Carry telemetry_map entry identity in-band so a re-created counter cannot be diffed against a dead baseline #290 replaced the whole inference with an in-band created_ns entry stamp (WAL v7) so no future deletion path can inherit the bug — the notification was removed as redundant. Live-validated three runs per configuration: 720.3 KiB ×3 unfixed vs 4.1 MiB ×3 fixed against a 3 MiB floor, and 4.1 MiB ×3 again with the notification removed, proving the stamp alone suffices. Detection gap also closed: the gc-pressure-relief scenario had passed five consecutive times on c36 against the defective agent because its GC watermarks only forced eviction on a busy map; they now truncate to zero entries, making eviction unconditional on any cluster. Knowledge landed in the handbook (kb/lachesis/known-issues/pressure-relief-eviction-unbills-busy-flows.md).

• 2026-07-14 — Bug [Bug] Ghost sweep re-buckets deleted VMs' historical bytes to unknown — per-tenant series non-monotonic, under-billing #111 CLOSED (all 7 requirements). The settled-bytes fold merged (telemetry PR 66) and its live regression landed as a step-scripted scenariotest scenario (telemetry PR 67): run mac-reuse validates monotonicity across the ghost sweep, no unknown re-bucketing, and clean MAC reuse across tenants — live PASS on staging, 23/23 rows, cross-run monotone through two sweeps. The new Scenario.Steps mechanism (open step vocabulary: drive/assert/capture/delete-vm/await-sweep/monotone/max-growth/boot-vm/sleep) is the base for further operational scenarios still to add (e.g. agent-down-for-N-seconds billing continuity); the story stays open for those and the remaining per-server-export AC.