From 6d0951ab17352f2fd7238189a85667be688753f3 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 25 Aug 2026 17:17:41 +1000 Subject: [PATCH 1/2] =?UTF-8?q?Reviews:=20audit=20run=202=20record=20?= =?UTF-8?q?=E2=80=94=20QuantEcon/meta,=20resumability=20validated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second run in the #16 validation program, and the first to test the claim the program exists to check: an interrupted run resuming across a lost session. It resumed at the predicted item with no re-walk, no skip and no duplicates. Two claims move from asserted to measured. Read-only is verified by a before/after fingerprint of all 317 issues rather than asserted. The open-count scaling conclusion gains a second data point. The principal method finding is invisible in the bundle and came from observing the run externally: the closed pass is not incrementally checkpointed — one write for 179 items against 15+ for 138 open — so SKILL.md's "loses one item rather than the phase" does not hold there. That is run 1's defect 2 in a new form, the 0.2.0 fix having addressed the format and not the granularity. Two candidate findings were investigated and discarded; both are recorded with their reasons, so the report shows its own false positive rate. Co-Authored-By: Claude Fable 5 --- reviews/audit-run-meta-2026-08-25.md | 247 +++++++++++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 reviews/audit-run-meta-2026-08-25.md diff --git a/reviews/audit-run-meta-2026-08-25.md b/reviews/audit-run-meta-2026-08-25.md new file mode 100644 index 0000000..36455ce --- /dev/null +++ b/reviews/audit-run-meta-2026-08-25.md @@ -0,0 +1,247 @@ +# Audit run 2 — `QuantEcon/meta`, 2026-08-25 + +Second run in the [`/qe:audit-issues` validation program](https://github.com/QuantEcon/skills/issues/16), and the +first run of the skill under its consolidated name (`qe` 0.7.0; run 1 was `audit` 0.1.x). Its headline purpose was +to falsify the one claim the program exists to check and run 1 left untested: **does an interrupted run resume +correctly across a lost session?** + +Run 1's record, for comparison throughout, is +[`audit-run-action-translation-2026-07-28.md`](audit-run-action-translation-2026-07-28.md). + +**Method note.** The run was executed by the maintainer in a Claude Code session. A second, independent session +observed it from outside — sampling `findings.md` every 15–20 seconds, fingerprinting the tracker before and +after, and recording a prediction of the resume point *before* the resume happened. That external observation is +what produced the findings in §5; none of them is visible in the finished bundle. Recommended for run 3. + +## 1. Snapshot provenance + +| Field | Value | +|---|---| +| Repo | `QuantEcon/meta` (public) | +| `snapshot_utc` | `2026-08-25T05:49:04+00:00` | +| `gh_version` | 2.98.0 (2026-08-20) | +| `fetched_by` | mmcky | +| Baseline ref | `origin/main` @ `58d577d` | +| Captured | 317 issues, 23 PRs, 340 items; highest number 368 | +| Open / closed | 138 / 179 | +| Unaccounted numbers | 28, **all 28 explained** (8 transferred to other repos, 20 converted to Discussions) | +| Truncation | none — `issues_at_limit: false`, `prs_at_limit: false` | + +The plan's stored figures (161 open / 319 items, measured 2026-08-07) had drifted; 138 / 317 is the measurement at +run time. Worth noting for estimates: a stored open-count ages faster than the total. + +## 2. Cost and duration + +| Measure | Value | +|---|---| +| Wall clock, snapshot → final bundle file | ~53 minutes | +| — of which | one operator-controlled interruption gap of unmeasured length | +| Resumed session, at the "closed batch D" mark | 19m 0s, 60.5k tokens | +| Open pass, observed rate | 138 items; 85 → 138 in 652s of the sampled window | +| Bundle | `README.md` 7.5K, `01-` 17K, `02-` 155K, `03-` 10K, plus `findings.md` 117K and `snapshot/` | + +Against run 1 — 230 items, 56 open, 22 minutes — this run was 317 items and 138 open in roughly 53 minutes. +Both the open count (2.5×) and the elapsed time (~2.4×) scaled together, which is consistent with run 1's +conclusion that *the open-issue count is what scales*, now supported by a second point rather than one. It is +still two points on a line, and this run's total includes an interruption, so it bounds rather than establishes +the rate. + +**The "expect hours, not minutes" claim that run 1 corrected stays corrected.** A 317-item tracker with 138 open +issues finished inside an hour including a full restart. + +## 3. Claims under test + +Each row is [#16's claims table](https://github.com/QuantEcon/skills/issues/16), marked against what this run +actually produced. + +| Claim | Verdict | Evidence | +|---|---|---| +| Resumes across a lost session | **HELD** | Resumed at `#261` — the predicted point, recorded before the fact. See §4. | +| Thread-complete on the closed side | **HELD** | All 179 closed threads read; 194 comments on open issues, 416 on closed, 131 comments + 10 review bodies on PRs. Five closed-side defects found that exist *only* in threads (§6). | +| Every claim carries an evidence class | **HELD** | 176 evidence tags across 138 catalogued open issues — 9 `[verified]`, 97 `[verified live]`, 61 `[stated]`, 9 `[inferred]`. No untagged status line found. | +| The self-audit closes gaps rather than disclosing them | **HELD** | 28 unaccounted numbers, each explained individually with its destination named, not waved at collectively. Residue enumerated in four numbered classes. | +| Read-only | **HELD — verified, not asserted** | See §4. | +| Sibling checks catch wave escapes | **HELD** | e.g. `#330` found fully delivered by checking all 17 `g4dn.2xlarge` workflow files across repos while its own table still marked four repos pending; `#261` found superseded by `#282` because seven of its nine "pending" repos no longer have a `gh-pages` branch. | +| Tiering slots into the repo's existing plan | **PENDING** | The run discovered `#344` and `#357` as plan anchors and tiered against them — but whether a maintainer *agrees* with the tiering is the one thing no self-audit can check. Awaiting mmcky's read; this is run 1's check 9/10 in a new instance. | + +**A note on `[verified]` in a repo with no code.** This run was chosen because `meta` has no code to verify +against — the sharpest stress on doctrine rule 1 ("the thread is a hypothesis and the default branch is the +evidence"), where `meta`'s default branch is two files. The audit's answer was to lean on +**`[verified live YYYY-MM-DD]`** — 97 of its 176 tags — checking claims against *other* repositories' live state, +published sites, and workflow files rather than against `meta` itself. That is a reasonable adaptation and it is +declared in the catalog's own legend. Whether the doctrine should name that as a first-class evidence class, +rather than leaving it to be improvised per run, is a real question for +[#12](https://github.com/QuantEcon/skills/issues/12) — and it is exactly the question #16 predicted this repo +would raise. + +## 4. Interruption log + +**State at kill** (captured externally before the resume, so it cannot be reconstructed after the fact): + +- Phase 2, **open pass**; no `## Closed` section existed yet +- 68 entries recorded, `#4` → `#260`, ascending, none outside the open partition +- The final write (`#260`) was **complete and clean** — all four fields present, file terminated with a newline +- `findings.md` 42,028 bytes + +**Prediction recorded before resuming**: resume at `#261` (the lowest open issue with no entry); do not re-walk +`#4`–`#260`; do not skip `#261`. + +**What actually happened**: + +| Expectation | Result | +|---|---| +| Resume at `#261` | Yes | +| No re-walking of completed items | Yes — the pre-kill 42,028 bytes were byte-identical afterwards; pure append | +| No skipping the item it died on | Yes | +| No duplicate entries | Yes — none anywhere in the file | +| Announce state correctly | Yes — "68 of the open issues are verified through `#260`; the closed pass hasn't started", matching an independent computation exactly | + +**Two things this interrupt did *not* test**, stated so run 3 can target them: + +1. **The truncation guard.** SKILL.md says to "re-verify the last entry in each rather than trusting a possibly + truncated write". The kill left a clean write, so the guard was never exercised. Whether `#260` was + re-verified is *indeterminate from the artifacts* — there is no duplicate entry, but re-verifying need not + mean rewriting. Only the session transcript could settle it. **Run 3 should interrupt mid-write** to produce + a genuinely partial entry. +2. **Resuming the closed pass.** The kill landed in the open pass. Given §5, resuming *inside* the closed pass is + now the more interesting test. + +**Read-only, verified rather than asserted.** A fingerprint of all 317 issues — number, state, `updated_at`, +comment count, labels — was taken before the run and again after: + +``` +before: 317 issues, sha256 024ccc8d4cd437e7… +after: 317 issues, sha256 024ccc8d4cd437e7… (identical) +``` + +Any close, label, or comment would have moved `updated_at` on the affected issue. The working tree was likewise +unchanged: `HEAD` still `58d577d`, `git status` showing only `?? .audit/`. Run 1 asserted read-only compliance; +this is the first run to demonstrate it. **Recommend the fingerprint-diff become a standard step** — it costs two +`gh` calls and converts a doctrine promise into a measurement. + +*Caveat, stated because it bounds the claim*: the baseline was taken shortly after the run began, not before it, +so a mutation in the first minutes would not have been caught. + +## 5. Finding — the closed pass is not incrementally checkpointed + +This is the run's principal method finding, and it is invisible in the finished bundle. + +Sampling `findings.md` throughout gave the write pattern: + +| Phase | Checkpoint writes | Granularity | +|---|---|---| +| Open (138 items) | 15+ | batches of 8–9 entries, every 75–120s | +| Closed (179 items) | **1** | nothing for the entire survey, then the complete 10.8 KB section in a single write | + +SKILL.md's phase 2 states the guarantee plainly: + +> Write each item's finding to `findings.md` as it is verified — **the closed pass too** … so that an interrupted +> run loses one item rather than the phase. + +For the closed side that does not hold. An interrupt anywhere in the closed survey loses **the phase** — all 179 +threads re-read from scratch. + +**This is run 1's defect 2 in a new form.** Defect 2 was "the closed side is never checkpointed"; `audit` 0.2.0 +added the `## Closed` heading and the two-partition resume rule. That fixed the *format* and left the +*granularity* untouched — the closed side is still all-or-nothing, now with a heading to be all-or-nothing under. + +There is a second-order consequence worth recording separately, because the two conventions were written +independently and do not compose: + +- **`deliverables.md` permits grouped entries** — "entries are grouped where a whole family shares one answer; + every closed issue is named exactly once" — and this run used them, giving 9 `**#N` headers covering all 179 + closed issues. +- **The resume rule addresses by issue number** — "resume at the lowest number with no entry under the matching + heading" — which is not well-defined against grouped prose. + +So even if the closed pass *were* checkpointed incrementally, the resume rule as written could not address into +it. Any fix needs to touch both. + +**What was not concluded, and why.** Two candidate findings were investigated and discarded rather than reported, +because the evidence did not support them: + +- *Out-of-order writes create a silent-skip hazard.* The post-resume block was written `261, 282, 263, 264, …` — + genuinely out of order. But the documented resume rule is a set difference ("lowest number with no entry"), + which is order-independent, so ordering is cosmetic. The hazard argument required assuming the run used a + high-water-mark algorithm, and the only evidence for that was it *describing* its position as "verified through + `#260`" — an accurate plain-English summary of a correct set-difference result. Inferring an implementation + from prose is not evidence. **Residue**: SKILL.md's guard names two wrong shortcuts ("do not infer progress + from the file's length or from a single block") but not "the highest recorded number". Adding the third is a + one-line clarification, not a defect. +- *Batched writes violate append-as-you-go.* They do not. Writing after 9 of 70 items is checkpointing + mid-phase, which is the doctrine's actual concern. Once measured, the exposure is bounded and modest — ~9 items + and ~2 minutes. Whether the doctrine should name a target granularity is a design question, not a compliance + failure. + +Both are recorded here because a validation report that only lists confirmed findings hides its own false +positive rate, and because the discipline that killed them — refusing a claim whose citation does not actually +support it — is the same discipline run 1's severity-1 defect failed. + +## 6. What the audit found about `meta` + +Reported as the audit's own output, not re-verified by this record. The maintainer read (§3, pending) is what +validates it. + +- **The tracker systematically understates what is already done** — five independent instances of one cause, + rollout state tracked in hand-maintained markdown tables that drift: `#330` (fully delivered, table shows four + repos pending), `#363`/`#364` (both books live, six unticked boxes describing shipped work), `#298` (targets + theme v0.20.2; theme is at v0.22.0), `#351` (two of three rows done), `#261` (superseded by `#282`). +- **19 of 138 open issues (14%) are verifiably closeable**, each with a drafted, unsent closing comment. +- **The urgent work is unlabelled and unlinked** — three confirmed reader-visible defects: `discourse.quantecon.org` + has no DNS record while five published pages still link to it; `intro.quantecon.org/inequality.html` contains no + `Plotly.newPlot` while its prose promises a figure; `dynamics.quantecon.org/black_litterman.html` ships draggable + sliders with no kernel. +- **Five closed-side defects**, all thread-only — most notably `#85`, closed COMPLETED in 2023 with half its scope + never done: no lecture repo carries a `LICENSE` file, confirmed by 404 on `GET /repos/{repo}/license` across six + repos. +- **Nine proposed new issues**, drafted ready to file. + +**Doctrine §5 compliance, observed live**: the run initially recommended attaching native sub-issues to `#362` and +`#335`, then a GraphQL check showed `#362` already complete and `#358` carrying eleven. It **folded the correction +into the documents** rather than appending it — which is what §5 requires — and narrowed its recommendation from +"adopt sub-issues" to "stop recording rollout state in issue bodies". + +## 7. What the doctrine did and did not transfer + +Per #16, three verdicts are distinct: a rule cited and load-bearing, a rule that never came up, and a rule the run +had to work around. Only the third is a bug. + +**Load-bearing.** §2's evidence classes (176 tags); §3's read-only boundary (verified clean); §5's +close-the-gap-don't-disclose-it (28/28 explained, and the folded correction above); `quantecon-context.md`'s +sibling-repo checks, which produced the single highest-value finding class. + +**Never came up.** The `.dev/` notes-system conventions — `meta` has no notes system, so the working directory +fell through to the documented `.audit/` fallback. The bundle-scale rule also did not bind at 138 open issues. + +**Worked around.** Two: + +1. **`[verified]` against a repo with no code**, resolved by improvising `[verified live ]` (§3). It worked, + it is declared, and it should probably be promoted into the doctrine rather than reinvented per run. +2. **Plan-anchor discovery with no `PLAN.md`** — `quantecon-context.md`'s search order fell through to *prior + audit issues* (`#344`, `#357`) as the anchor. That is arguably the intended behaviour, but the search order + does not name it explicitly, and it is now the second repo where the notes-system-first ordering did not apply. + +## 8. Recommended follow-ups + +Filed separately, not as paragraphs here, per the plan convention: + +1. **Checkpoint the closed pass incrementally**, and reconcile the resume rule with grouped entries (§5). This is + the one defect of substance. +2. **Promote `[verified live ]` into `doctrine.md` §2** as a named evidence class, or state explicitly that + ref-relative verification does not apply to trackers without code (§3, §7). +3. **Add "the highest recorded number" to SKILL.md's list of wrong ways to infer progress** (§5, residue). +4. **Make the tracker fingerprint-diff a standard run step** (§4). +5. **Run 3 should interrupt mid-write, and inside the closed pass** (§4). +6. **`#23` input**: the closed pass is 179 of 317 items — 56% of the run — for the cheapest checks in the audit. + It scales badly (on `lecture-python.myst`, 1003 items against 73 open, it would be ~93% of the run) and, unlike + the bundle, has no scale rule. Whether it earns that share is now answerable with two data points. + +## 9. Outstanding — the maintainer's read + +The one thing no self-audit can supply, and run 1's checks 9 and 10 in a new instance: + +- Does the tiering against `#344`/`#357` match how the repo is actually planned, and would you act on it? +- Are the 19 drafted closing comments ones you would send? +- Where should the bundle live? It contains AWS and access-token findings; the run flagged + `QuantEcon/infrastructure` as the better home for those portions. `meta` is public and has no notes system, so + the bundle currently sits untracked in `.audit/`. From 53c2f0520607f4c0a4d7f749fee69348ffd55a59 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 25 Aug 2026 17:27:36 +1000 Subject: [PATCH 2/2] Reviews: pin run-2 citations to qe--v0.7.0, sharpen the read-only wording MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's review of #56 caught two things. Quoted guarantees pointed at a bare "SKILL.md" with no repo, path or version, so they would stop resolving the moment the skill changed — the same class of defect run 1 was faulted for, one level up. All six references now point at the tag whose tree this run actually executed. And "the working tree was unchanged" sat next to a git status showing ?? .audit/, which reads as a contradiction. It now says no tracked file changed, with the untracked output directory named as expected. Co-Authored-By: Claude Fable 5 --- reviews/audit-run-meta-2026-08-25.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/reviews/audit-run-meta-2026-08-25.md b/reviews/audit-run-meta-2026-08-25.md index 36455ce..e3b96b0 100644 --- a/reviews/audit-run-meta-2026-08-25.md +++ b/reviews/audit-run-meta-2026-08-25.md @@ -13,6 +13,10 @@ observed it from outside — sampling `findings.md` every 15–20 seconds, finge after, and recording a prediction of the resume point *before* the resume happened. That external observation is what produced the findings in §5; none of them is visible in the finished bundle. Recommended for run 3. +**Citations are pinned to `qe--v0.7.0`** — the tag whose tree this run actually executed — so every quoted +guarantee stays checkable against the same artifact even after the skill changes. A run record whose citations +drift is the defect run 1 was faulted for, one level up. + ## 1. Snapshot provenance | Field | Value | @@ -98,7 +102,7 @@ would raise. **Two things this interrupt did *not* test**, stated so run 3 can target them: -1. **The truncation guard.** SKILL.md says to "re-verify the last entry in each rather than trusting a possibly +1. **The truncation guard.** [`SKILL.md`](https://github.com/QuantEcon/skills/blob/qe--v0.7.0/qe/skills/audit-issues/SKILL.md) says to "re-verify the last entry in each rather than trusting a possibly truncated write". The kill left a clean write, so the guard was never exercised. Whether `#260` was re-verified is *indeterminate from the artifacts* — there is no duplicate entry, but re-verifying need not mean rewriting. Only the session transcript could settle it. **Run 3 should interrupt mid-write** to produce @@ -114,8 +118,9 @@ before: 317 issues, sha256 024ccc8d4cd437e7… after: 317 issues, sha256 024ccc8d4cd437e7… (identical) ``` -Any close, label, or comment would have moved `updated_at` on the affected issue. The working tree was likewise -unchanged: `HEAD` still `58d577d`, `git status` showing only `?? .audit/`. Run 1 asserted read-only compliance; +Any close, label, or comment would have moved `updated_at` on the affected issue. **No tracked file changed** +either: `HEAD` still `58d577d` and a clean index with no diff, the only entry in `git status` being the +untracked `?? .audit/` — the run's own output directory, which is where the working-directory rule puts it. Run 1 asserted read-only compliance; this is the first run to demonstrate it. **Recommend the fingerprint-diff become a standard step** — it costs two `gh` calls and converts a doctrine promise into a measurement. @@ -133,7 +138,7 @@ Sampling `findings.md` throughout gave the write pattern: | Open (138 items) | 15+ | batches of 8–9 entries, every 75–120s | | Closed (179 items) | **1** | nothing for the entire survey, then the complete 10.8 KB section in a single write | -SKILL.md's phase 2 states the guarantee plainly: +[`SKILL.md`](https://github.com/QuantEcon/skills/blob/qe--v0.7.0/qe/skills/audit-issues/SKILL.md) phase 2 states the guarantee plainly: > Write each item's finding to `findings.md` as it is verified — **the closed pass too** … so that an interrupted > run loses one item rather than the phase. @@ -148,7 +153,7 @@ added the `## Closed` heading and the two-partition resume rule. That fixed the There is a second-order consequence worth recording separately, because the two conventions were written independently and do not compose: -- **`deliverables.md` permits grouped entries** — "entries are grouped where a whole family shares one answer; +- **[`deliverables.md`](https://github.com/QuantEcon/skills/blob/qe--v0.7.0/qe/references/audit/deliverables.md) permits grouped entries** — "entries are grouped where a whole family shares one answer; every closed issue is named exactly once" — and this run used them, giving 9 `**#N` headers covering all 179 closed issues. - **The resume rule addresses by issue number** — "resume at the lowest number with no entry under the matching @@ -165,7 +170,7 @@ because the evidence did not support them: which is order-independent, so ordering is cosmetic. The hazard argument required assuming the run used a high-water-mark algorithm, and the only evidence for that was it *describing* its position as "verified through `#260`" — an accurate plain-English summary of a correct set-difference result. Inferring an implementation - from prose is not evidence. **Residue**: SKILL.md's guard names two wrong shortcuts ("do not infer progress + from prose is not evidence. **Residue**: [`SKILL.md`](https://github.com/QuantEcon/skills/blob/qe--v0.7.0/qe/skills/audit-issues/SKILL.md)'s guard names two wrong shortcuts ("do not infer progress from the file's length or from a single block") but not "the highest recorded number". Adding the third is a one-line clarification, not a defect. - *Batched writes violate append-as-you-go.* They do not. Writing after 9 of 70 items is checkpointing @@ -227,9 +232,9 @@ Filed separately, not as paragraphs here, per the plan convention: 1. **Checkpoint the closed pass incrementally**, and reconcile the resume rule with grouped entries (§5). This is the one defect of substance. -2. **Promote `[verified live ]` into `doctrine.md` §2** as a named evidence class, or state explicitly that +2. **Promote `[verified live ]` into [`doctrine.md`](https://github.com/QuantEcon/skills/blob/qe--v0.7.0/qe/references/audit/doctrine.md) §2** as a named evidence class, or state explicitly that ref-relative verification does not apply to trackers without code (§3, §7). -3. **Add "the highest recorded number" to SKILL.md's list of wrong ways to infer progress** (§5, residue). +3. **Add "the highest recorded number" to [`SKILL.md`](https://github.com/QuantEcon/skills/blob/qe--v0.7.0/qe/skills/audit-issues/SKILL.md)'s list of wrong ways to infer progress** (§5, residue). 4. **Make the tracker fingerprint-diff a standard run step** (§4). 5. **Run 3 should interrupt mid-write, and inside the closed pass** (§4). 6. **`#23` input**: the closed pass is 179 of 317 items — 56% of the run — for the cheapest checks in the audit.