Add single-writer helper + schema validation for tick-history and archive metadata

[tomerweller]

Closes #3791

Summary

The monitor's per-tick artifacts had fragmented into many incompatible schemas because each tick path (interactive vs headless) hand-rolled its own row/heredoc — tick-history.jsonl had 100+ key-signatures and warnings accumulated 571 spellings for ~17 real conditions (#3757 census). This PR introduces scripts/lib/monitor-tick-artifacts.py as the single writer of every per-tick artifact and replaces the four hand-rolled heredocs in monitor-tick/SKILL.md with calls to it, so every appended row is conformant by construction.

Key pieces:

• emit-row builds the canonical 9-field row (corrected set: no tick, with self_reflect), stamps a UTC ISO-8601 ts, constrains warnings/actions to a closed vocabulary (unknown → registered other fallback), promotes embedded measurements to typed sibling keys (low-disk 90% → warnings:["low-disk"], disk_free_pct:90), and self-checks before append. Self-check failure is fail-loud-but-non-fatal: the row goes to a tick-history.rejected.jsonl sidecar and the tick continues, so a helper bug can never wedge the monitor.
• validate-row enforces a required-subset + per-field-type contract (never set-equality, so diagnostic supersets pass — the "canonical-9 equality rejects 100% of the corpus" trap) and type-guards ts before the regex (null/int ts rejected without raising).
• Vocabulary = the 37 alarm names in metric-alarms.toml ∪ a new top-level [warning_vocabulary].non_alarm table ∪ {other}. eval-alarms.py reads only [[alarm]]/schema_version, so the new table is inert (catalog still validates: alarm_count=37, valid=true).
• Archive metadata: .prom pair copied with cp -p so archived mtimes describe the archived bytes (prune stays mtime-based); ARCHIVE_VERSION is now a meaningful, helper-owned constant (2) instead of the inert 1.

Plan reference

Converged Plan comment

Test plan

• scripts/test-monitor-skill-snippets.sh — 477/477 pass (was 465; +12 new monitor: single-writer helper + schema validation for tick-history.jsonl and archive metadata.env #3791 assertions, Tests 64/65 rewritten)
• cargo fmt --check (no Rust changed)
• python3 -m py_compile scripts/lib/monitor-tick-artifacts.py
• eval-alarms.py --validate-only on the catalog with the new table: valid=true
• End-to-end simulation of the SKILL tick-history snippet produces a valid canonical row (measurement promotion, vocabulary normalization, filed-#N preserved, watch retained)

Note: --strict mode reports a pre-existing structural-drift warning in monitor-loop/SKILL.md (unrelated file, present on origin/main); the default suite is green.

Regression test (bug-fix)

• Tests: scripts/test-monitor-skill-snippets.sh — 14 assertions (rewritten Tests 64/65 + 12 new; e.g. tick-row validate: null ts → clean reject, tick-row emit: embedded measurement promoted, archive metadata: ARCHIVE_VERSION=2).
• Pre-fix: committed as c04cdee — verified all 14 FAILED on main (helper absent → emit-row/validate-row error; SKILL still hand-rolls the heredoc).
• Post-fix: verified all PASS after 8c4d184.

Deviations from plan

• Tick-history happy-path append is written as if ! ROW_ERR=$(... 2>&1 1>>"$HIST"); then <sidecar> (capturing stderr without a temp file) rather than a bare emit-row ... >> "$HIST", so the reject-sidecar path is wired inline while keeping the stdout-row/append contract. No behavior change on the happy path.
• actions vocabulary is a small fixed verb set kept in the helper (with filed-#<N> normalizing to filed while preserving the full token) rather than a second TOML table — the plan specified only one new [warning_vocabulary] table.

🤖 Generated with Claude Code

[tomerweller]

🔍 Reviewer: Correctness

Verdict: CHANGES_REQUESTED

Summary: The single-writer helper is well-structured and conformant-by-construction, but CI is red (the .agents/ catalog mirror was not synced) and a shipped consumer of the archive metadata still hard-rejects the new ARCHIVE_VERSION=2.

Full review

Cycle 1 — complete class-labeled change list. A class not raised here should not be raised in a later cycle.

[ci-failure / mirror-drift] — BLOCKING
scripts/lib/test_eval_alarms_gauge_guard.py::test_agents_catalog_matches_claude FAILED in the "Script self-tests" check (run 32668696211). The test asserts .agents/skills/shared/metric-alarms.toml is byte-identical to the .claude/ copy. This PR added the 31-line [warning_vocabulary] block only to .claude/skills/shared/metric-alarms.toml; the .agents/ mirror was left unchanged, so the copies diverged.

• Fix: apply the identical [warning_vocabulary] block to .agents/skills/shared/metric-alarms.toml so the two files are byte-for-byte identical. (The alarm-version-check workflow watches .agents/skills/shared/**, confirming this mirror is gated.)

[regression-risk] — BLOCKING
scripts/dev/replay-alarms-on-history.sh:137-140 hard-rejects any archive whose metadata.env has ARCHIVE_VERSION != "1":

if [[ "${ARCHIVE_VERSION:-}" != "1" ]]; then
  echo "ERROR: Corrupt archive at $dir: ARCHIVE_VERSION=${ARCHIVE_VERSION:-missing} (expected 1)" >&2
  return 1

This PR bumps the constant to ARCHIVE_VERSION=2 in the new write-metadata subcommand. After this ships, every new archive snapshot the monitor writes carries ARCHIVE_VERSION=2, and the replay/regression tooling (replay-alarms-on-history.sh, and check-alarm-regression.sh which drives it) will treat all of them as "Corrupt archive ... (expected 1)" and refuse to replay them. Making the version "meaningful" requires the reader to branch on it, not fail-closed on anything but 1.

• Fix: update replay-alarms-on-history.sh (and any other reader) to accept the known-good set (e.g. 1 or 2), and add a test that a v2 archive is accepted. This is not covered by the current CI, so a regression test belongs in the same PR.

[test-coverage] — non-blocking note
The new emit-row/validate-row/write-* paths are well covered by the snippet tests (12 new TAP assertions, plan 465→477). Good. But there is no test asserting the archive-version reader tolerates the bumped version (see the regression above) — add one alongside the fix.

Nits (inline, non-blocking):

• build_row: for a warning token that carries an embedded number whose base is NOT in MEASUREMENT_KEYS (e.g. pruning-gap 5), the number is silently dropped while the base token is kept. Probably fine, but worth a comment that only mapped bases preserve the measurement.
• validate_row restricts deploys to {0,1}. If a tick ever performs 2 deploys the row is routed to the reject sidecar. The SKILL template says <0 or 1>, so acceptable, but the constraint is stricter than "a count".

[tomerweller]

🔍 Reviewer: Risk

Verdict: CHANGES_REQUESTED

Summary: Non-parity change (skills/scripts only — no observable core surface). Two blocking risks: CI is red from an un-synced catalog mirror, and the ARCHIVE_VERSION bump breaks the archive replay tooling that reads it.

Full review

Cycle 1 — complete class-labeled change list.

[ci-failure] — BLOCKING
CI "Script self-tests" is red: test_agents_catalog_matches_claude — catalog copies diverged. The .agents/skills/shared/metric-alarms.toml mirror must be updated to match the .claude/ copy byte-for-byte. Diff-attributable (this PR edited the .claude/ catalog).

[regression-risk] — BLOCKING
scripts/dev/replay-alarms-on-history.sh:137 rejects archives unless ARCHIVE_VERSION == "1". This PR emits ARCHIVE_VERSION=2. Result: the replay/regression harness rejects every archive produced after this ships. The version bump and its reader must land together. Update the reader to accept {1,2} and add coverage.

[operational] — non-blocking
emit-row/validate-row require tomllib (Python 3.11+) or a tomli fallback to load the vocabulary. If the monitor host's python3 is <3.11 without tomli, load_warning_vocab raises, emit-row exits non-zero, and every row is routed to tick-history.rejected.jsonl — fail-safe (tick continues) but silently produces zero canonical rows. Confirm the monitor host ships Python ≥3.11 (CI uses 3.12).

Parity note: none of the touched paths (.claude/skills/**, scripts/**, metric-alarms.toml) are on the observable/interop surface per docs/PARITY.md. No ledger/bucket hash, tx result/meta XDR, SCP/overlay wire, history-archive-format, or RPC/CLI contract is affected. No parity concern.

[tomerweller]

Review: Bounce-Back Cycle 1

Reason: Reviewer A + Reviewer B requested changes; CI failed (diff-attributable).

Reviewer A (Correctness): CHANGES_REQUESTED — .agents/ catalog mirror not synced (CI red) + ARCHIVE_VERSION=2 breaks the archive replay reader.
Reviewer B (Risk): CHANGES_REQUESTED — same two blockers; non-parity change, no observable-surface concern.
External reviewers: none.
CI: red — test_agents_catalog_matches_claude FAILED ("catalog copies diverged") in the "Script self-tests" check.

Two blocking findings for /do Mode B to address (both survive refutation; finding 1 is the CI failure itself, finding 2 is a verified code path):

1. [ci-failure / mirror-drift] Sync .agents/skills/shared/metric-alarms.toml to be byte-identical to .claude/skills/shared/metric-alarms.toml (add the same [warning_vocabulary] block). This is what turns CI green.

2. [regression-risk] scripts/dev/replay-alarms-on-history.sh:137-140 rejects any archive with ARCHIVE_VERSION != "1" as corrupt, but this PR emits ARCHIVE_VERSION=2. Update the reader (and check-alarm-regression.sh if affected) to accept {1,2}, and add a regression test that a v2 archive is accepted.

Failed-check excerpt:

scripts/lib/test_eval_alarms_gauge_guard.py::test_agents_catalog_matches_claude FAILED
E   AssertionError: catalog copies diverged
1 failed, 152 passed

Routing back to ready-for-doing for /do Mode B.

[tomerweller]

Addressed both blocking findings from Bounce-Back Cycle 1 in 5d22fe1:

1. [ci-failure / mirror-drift] Synced .agents/skills/shared/metric-alarms.toml byte-for-byte with the .claude/ copy (added the [warning_vocabulary] block). test_agents_catalog_matches_claude now passes locally (16/16 gauge-guard tests green).
2. [regression-risk] replay-alarms-on-history.sh now accepts the known-good version set {1,2} instead of fail-closing on anything but 1, so post-#3791 archives (ARCHIVE_VERSION=2) are no longer rejected as corrupt. check-alarm-regression.sh only drives the replay script and reads no version itself, so no other reader needed changing. Added a regression test (validate_metadata accepts ARCHIVE_VERSION=2) — verified it fails on the pre-fix reader and passes after — and bumped TAP_PLAN 477→478. Full snippet suite: 478/478 green.

Note: the unrelated CI / Test red (henyey-ledger manager::tests::test_scan_and_merge_pending_buffer_bounded) is not diff-attributable to this skills/scripts-only PR; it will re-run on this push.

Ready for re-review.

[tomerweller]

🔍 Reviewer: Correctness

Verdict: APPROVE

Summary: Cycle 2 re-review. Both cycle-1 blocking findings (ci-failure/mirror-drift, regression-risk) are resolved on the current head (5d22fe1); CI is fully green (43/43), the single-writer helper is clean and well-covered, and the requested regression test for the bumped archive version was added.

Full review

Cycle 2 — sticking to the classes raised in cycle 1. No NEW CLASS DISCOVERED.

[ci-failure / mirror-drift] — RESOLVED
.agents/skills/shared/metric-alarms.toml and .claude/skills/shared/metric-alarms.toml are now byte-for-byte identical (diff clean). The [warning_vocabulary] block was mirrored to .agents/. All 43 CI checks are green, including "Script self-tests" (which runs test_agents_catalog_matches_claude).

[regression-risk] — RESOLVED
scripts/dev/replay-alarms-on-history.sh now branches on a known-good set — case "${ARCHIVE_VERSION:-}" in 1|2) ;; — instead of fail-closing on anything but 1. Post-#3791 archives (ARCHIVE_VERSION=2) are accepted; the error message updated to "expected 1 or 2". check-alarm-regression.sh only drives the replay script and reads no version itself, so no other reader needed changing.

[test-coverage] — RESOLVED (was the non-blocking cycle-1 note)
A regression test was added — archive: validate_metadata accepts ARCHIVE_VERSION=2 (snippet suite line ~6169), plus archive metadata: ARCHIVE_VERSION=2 (meaningful, not inert 1). TAP plan bumped 477→478; full suite 478/478 green locally. Verified independently:

• emit-row happy path produces the canonical 9-field row with measurement promotion (low-disk 90% → warnings:["low-disk"], disk_free_pct:90), other fallback for unknown tokens, and filed-#42 preserved.
• The SKILL's reject-sidecar wiring (2>&1 1>>"$HIST") works: a self-check failure appends nothing to $HIST, routes the offending row to tick-history.rejected.jsonl, and the tick continues (fail-loud-but-non-fatal).
• write-metadata emits clean sourceable key=value lines with the full 11-field set matching the old heredoc; the replay reader sources it cleanly.

Verdict: all cycle-1 concerns addressed; no new correctness concerns.

[tomerweller]

🔍 Reviewer: Risk

Verdict: APPROVE

Summary: Non-parity change (skills/scripts/catalog only — no observable core surface per docs/PARITY.md). Both cycle-1 blocking risks (ci-failure, regression-risk) are resolved; CI is green. The one remaining item (operational — tomllib on Python <3.11) was non-blocking in cycle 1 and remains so.

Full review

Cycle 2 — sticking to the classes raised in cycle 1. No NEW CLASS DISCOVERED.

[ci-failure] — RESOLVED
Catalog mirror synced byte-for-byte; "Script self-tests" and all 42 other checks green.

[regression-risk] — RESOLVED
The archive-version reader (replay-alarms-on-history.sh) now accepts {1,2} and lands together with the ARCHIVE_VERSION=2 bump, with a regression test proving a v2 archive is accepted. No archive written after this ships will be rejected as corrupt.

[operational] — non-blocking (unchanged from cycle 1)
load_warning_vocab uses tomllib (3.11+) with a tomli fallback. On a host with Python <3.11 lacking tomli, emit-row would exit non-zero and every row would route to the reject sidecar — fail-safe (tick continues) but silently produces zero canonical rows. CI runs 3.12. Not a blocker; noted for operator awareness.

Parity note: none of the touched paths (.claude/skills/**, .agents/skills/**, scripts/**, metric-alarms.toml) are on the observable/interop surface. No ledger/bucket hash, tx result/meta XDR, SCP/overlay wire bytes, history-archive format, or RPC/CLI contract is affected. No parity concern.

[tomerweller]

✅ Merged

Commit: ebbe574

Triple-green (cycle 2): Reviewer A (Correctness) APPROVE, Reviewer B (Risk) APPROVE, CI green (43/43). Both cycle-1 blocking findings (catalog mirror-drift, ARCHIVE_VERSION reader) confirmed resolved on the current head.

Follow-up issues filed for unaddressed inline review comments: none (no inline threads).