perf(converter): read each session once for the converter and the audit - #9
Merged
Merged
Conversation
convert_and_audit and convert_codex_and_audit read every transcript file once. raw_records.load_session stats a file, then streams its bytes through a sha256 digest into the JSON parser, so the fingerprint is the fingerprint of the bytes that were parsed. The converter, the census, the edges emitter and the enrichment pass consume that one list of records, and the fingerprints are re-checked (stat and digest) once the artifacts are built. Before: each file was opened five times (snapshot hash, converter read, re-check hash, audit parse, re-check hash), parsed twice and hashed three times with blake2b. After: two opens, one parse, two sha256 passes, which run at 1.6 GB/s on this host's SHA extensions against blake2b's 0.9 GB/s. Measured on the 76 MB benchmark session (59 MB of side files): the use case went from 1.55 s to 1.01 s in-process; `atif-sql convert` from 2.78 s / 797 MB to 2.07 s / 736 MB (median of 5); a materialize of the 300-session Claude snapshot from 13.7 s to 11.6 s wall with 49.7 s to 34.8 s of summed worker convert time. Byte identity: 378 sessions across both frozen corpora, 0 differing trajectory.json, edges.jsonl, stdout summaries or exit codes; the live parity oracle passes for both agents at ATIF_PARITY_LIMIT=0. The same-size-rewrite-inside-one-mtime-tick guard is kept: the post conversion re-check still hashes, so a rewrite the stat pair cannot see still refuses the session. TestSinglePass pins the open, parse and hash counts for both agents and fails on the previous flow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Convert reads each transcript once
Climb run 2, attempt 1 (objective 1.1 against main
31b61d5). Every transcript file was opened 5 times, parsed twice and hashed 3 times across the snapshot, the converter, the audit census and two mutation re-checks. Now one pass reads, hashes (sha256 through a digesting reader under the same UTF-8 text layer) and parses each file, the converter and the audit share the resultingLoadedSession, and one mutation re-check runs after conversion. Codex gets the same treatment.convertof the largest session (76 MB), median of 5materialize --forceof the 300-session snapshot, 8 workersByte identity: live parity oracle over both frozen corpora, 0 divergences; direct compare against main's binary for all 378 sessions, 0 differing trajectory.json, edges.jsonl, stdout summaries or exit codes; the materialized snapshot's 600 artifacts identical.
Kept: the post-conversion re-check still hashes, because the pinned same-size-rewrite-within-one-mtime-tick test requires it (0.10 s of the 1.0 s in-process total). New
TestSinglePasstests pin opens, parses and digests per file and fail on main's code.mise run checkgreen (1403 passed, 1 skipped);mise run docs:gategreen.