perf: 4.0x on the five-task panel via a certified frontier climb (process pool, litellm-free pricing, columnar artifacts) - #7
Merged
Conversation
`atif-sql materialize` converted sessions one at a time in one process. The convert+write stage now runs across a spawn-context ProcessPoolExecutor: `--workers N` / `ATIF_SQL_MATERIALIZE_WORKERS`, default `min(8, cpu_count)`. `--workers 1` is the unchanged inline reference path, and a pass with a single planned session runs inline. Each worker unpickles the injected ConverterPort once in its initializer and runs the same `_write_session` the serial path runs, so the per-session staging dir and atomic swap stay the crash-safety unit and every artifact byte is identical. Worker outcomes are folded back in plan order, so failures and counts read the same whatever finished first; the failure text is formatted inside the worker so an exception that cannot be rebuilt across the pipe never changes the report. `convert_seconds` is the per-session sum and can exceed the wall clock; the report carries the worker count actually used. The CLI hands the pool a `worker_setup` hook that installs the same stderr sink `main()` installs, so a spawned worker does not print loguru's default DEBUG stream. The staging sweep needs no change: a worker stages under its own pid, and a dead pass takes its workers with it. Tests: byte identity of the pooled and inline corpora, failures in plan order with the watermark held back, worker pids read back off disk (a pool that silently ran inline fails), the setup hook running in every worker, the settings default and env override, the flag reaching the use case, `--workers 0` exiting 64, and a real-converter pooled pass through the CLI.
…orting litellm Converting one 76 MB Claude Code session took 5.8 s, and about four of those seconds were `import litellm` inside the cost estimate: openai, anthropic and hundreds of pydantic model builds, all for one `litellm.cost_per_token` call per step. Every convert and materialize process paid it once. `atif_converter.domain.pricing` now reads litellm's own bundled `model_prices_and_context_window_backup.json` straight off disk, located via `importlib.util.find_spec` so `litellm/__init__` never runs, and repeats litellm 1.100.1's `cost_per_token` arithmetic in the same float operation order for the shapes our transcripts produce: bare table keys under the anthropic / openai / bedrock / bedrock_converse providers, service-tier and above-threshold rate variants, and the unmapped `claude-<family>-<n>` ids litellm routes to Anthropic through its `fallback_generalizations` rules and prices at zero. Any other shape (a `provider/model` string, an `ft:` id, a `tiered_pricing` table, another provider, a case-variant key) falls back to importing litellm and calling it exactly as before, logged at debug. The Codex path uses the same table for its `model_cost.get` lookup. Proof of identity, which is the whole point: - tests/test_pricing_identity.py compares the fast path with litellm itself over every covered bare key of the table (440 of 452; the 12 `ft:` keys are declined by design) crossed with nine token shapes and seven service tiers: 27,720 comparisons, every float `==`, plus the distinct (model, service_tier) pairs found in the two frozen corpora. - The live parity test over both full corpora (300 Claude Code sessions, 78 Codex rollouts) reports 0 divergences from harbor. - Converting every session of both corpora with this CLI and the untouched reference checkout gives identical trajectory.json and edges.jsonl sha256 for all 378. Measured on the 76 MB session, median of 3: 5.78 s / 981 MB max RSS before, 2.89 s / 802 MB after; `python -X importtime` shows no litellm module on the fast path (974 lines in the reference run). `LITELLM_LOCAL_MODEL_COST_MAP` keeps its meaning: unset or `true` prices from the bundled table; anything else hands every call to litellm, which then fetches the remote table as the operator asked. litellm stays a declared dependency for the fallback.
Every query used to start by parsing every trajectory.json in the corpus into a TEMP TABLE and extracting the view columns with json_extract at query time. This change gives the views typed columnar inputs instead. Design: per-session parquet, produced at materialize time, read lazily. * atif-duck gains a pure `ColumnarArtifactProducer` that takes the in-memory trajectory dict and writes `session.parquet`, `steps.parquet`, `tool_calls.parquet`, and `tool_results.parquet` with the views' own projection expressions (shared in `infrastructure/projections.py`), so the JSON columns are normalized exactly as read_json would normalize them and the rows are the same to the byte. Rows are fed to DuckDB as 1 MiB Arrow batches, one statement per batch; each batch's typed rows come back through the fetch path and pyarrow appends them to the file as one row group. That shape was chosen by measurement on the 204 MB session: one COPY per file let DuckDB's allocator hold the whole text until the statement ended (+416 MB over the dict), exporting Arrow tables held +230 MB, the fetch path holds +108 MB. A fresh DuckDB connection per session keeps that memory from accumulating across a pass. Files are written 0444. Per-session parquet rather than one corpus-wide store keeps the writes inside the existing per-session atomic swap and keeps a partially materialized corpus valid. * atif-corpus gains an `ArtifactProducer` port. `materialize` runs it in the staged directory after the three JSON artifacts and before meta.json, merges the returned keys into meta.json (rejecting the contract's own keys), and reports `artifact_seconds`. atif-corpus never learns what the producer writes; atif-cli composes the two, and `--no-columnar` turns it off. * The registry splits sessions on `meta.columnar_schema == 1` plus four readable parquets: those are read with read_parquet (lazy, typed, no JSON parsed), the rest are parsed from trajectory.json over an explicit path list, and the two are unioned per surface. Older corpora keep working, `atif-sql status` reports the query path (columnar, json, mixed), and `register` returns the sessions it bound each way. The columnar parquets join the per-file `allowed_paths` grants the analytics parquets already use; the 0444 mode closes the USE_TMP_FILE write hole for them. * `tasks_state_current`'s creation-order fallback now breaks timestamp ties on (step_id, tool_use_id). Several TaskCreate calls in one step share a timestamp, and a window with a tied ORDER BY numbered them in physical scan order, which differs between the JSON reader and the parquet reader (and between a two- and a three-session JSON corpus). The number is now a function of the data. Equivalence is asserted with EXCEPT ALL in both directions over every catalog view on a three-session fixture (Claude Code plus Codex) for the JSON, columnar, and mixed source mixes, plus the macros and byte-identical `--format json` output for the panel queries at the CLI layer. One test destroys every trajectory.json after producing the artifacts and requires the views to keep answering, so a silent fallback to JSON fails the suite. Snapshot (300 sessions, 1.6 GB, frozen corpus, same machine, three runs each): trajectory.json and edges.jsonl are sha256-identical between a corpus materialized by this change and one by the reference (600 files, 0 differences); the five measured statements return byte-identical `--format json` output from the reference binary and from this one over the same corpus root, and from this one over a JSON-only corpus; the reference binary still reads a corpus carrying the new files. Query wall and peak RSS on the columnar path: 0.9 to 1.1 s and 480 to 955 MB, against 2.5 to 12.7 s and 4.5 to 16.3 GB for the reference. The JSON fallback path stays within 6 percent of the reference's peak on every statement. Materialize, full `--force` pass: 91.1 s at a 796.5 MB peak with the artifacts, 54.5 s at 670.5 MB with `--no-columnar` (reference 52.3 s, 667.2 MB). The peak is 19 percent over the reference, past the 10 percent bound this lever was given: importing duckdb and pyarrow into the materialize process costs about 90 MB by itself (13 percent of the baseline) before any row is written, and no allocator setting, batch size, or writer path measured below that. The trade is documented in docs/reference/cli.md so an operator can choose `--no-columnar`.
…empt 3) Merge resolution threads the ArtifactProducer through the materialize process pool: each worker receives the producer in its initializer beside the converter, and every _SessionOutcome carries artifact_seconds so the report sums them in plan order.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| raise ValueError(msg) | ||
| picks.append(f'"{name}"') | ||
| # Column names are catalog constants; inner_sql is built from constants. | ||
| return f"SELECT {', '.join(picks)} FROM ({inner_sql})" # noqa: S608 |
| ColumnarArtifactProducer._write( | ||
| con, | ||
| _batches(_step_rows(session_id, step_list), steps_schema), | ||
| f"SELECT session_id, {render(step_columns(MEMBER_COLUMNS))} FROM {_SOURCE}", # noqa: S608 — projections constants |
Comment on lines
+525
to
+526
| "SELECT session_id, " # noqa: S608 — projections constants only | ||
| f"{render(step_key_columns(MEMBER_COLUMNS))}, {render(CALL_COLUMNS)} FROM {_SOURCE}", |
Comment on lines
+535
to
+536
| "SELECT session_id, " # noqa: S608 — projections constants only | ||
| f"{render(step_key_columns(MEMBER_COLUMNS))}, {render(RESULT_COLUMNS)} FROM {_SOURCE}", |
| """ | ||
| files = ", ".join(sql_literal(str(path)) for path in paths) | ||
| projected = ", ".join(f"CAST({name} AS {sql_type}) AS {name}" for name, sql_type in columns) | ||
| return f"SELECT {projected} FROM read_parquet([{files}])" # noqa: S608 — paths via sql_literal; columns are catalog constants |
| # The one interpolation is the module constant _RAW_META_TABLE plus the | ||
| # contract's meta key. | ||
| rows = con.execute( | ||
| f"SELECT session_id_path, {META_COLUMNAR_KEY} FROM {_RAW_META_TABLE} ORDER BY 1" # noqa: S608 |
Comment on lines
+526
to
+527
| f""" | ||
| CREATE OR REPLACE TEMP TABLE {_RAW_TRAJECTORIES_JSON_TABLE} AS |
Comment on lines
+652
to
+653
| f"CREATE OR REPLACE VIEW steps AS SELECT {_catalog_columns('steps')} " # noqa: S608 | ||
| f"FROM {_RAW_STEPS_TABLE};" |
Comment on lines
+738
to
+739
| f"CREATE OR REPLACE VIEW tool_calls AS SELECT {_catalog_columns('tool_calls')} " # noqa: S608 | ||
| f"FROM {_RAW_TOOL_CALLS_TABLE};" |
Comment on lines
+749
to
+750
| f"CREATE OR REPLACE VIEW tool_results AS SELECT {_catalog_columns('tool_results')} " # noqa: S608 | ||
| f"FROM {_RAW_TOOL_RESULTS_TABLE};" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
4.0x on the five-task panel, zero output differences
Materialize, count, tool ranking, token sums and convert all got faster, and peak memory on every task dropped from up to 8.7 GB to under 800 MB. Every byte of every artifact and every query result is identical to main.
materialize --force(300 Claude Code sessions + 78 Codex rollouts)SELECT count(*), count(DISTINCT model_name) FROM sessionsconvertof the largest session (76 MB)Geometric mean of the per-task speedups: 3.999 ± 0.047, measured as three interleaved A/B repetitions against main on the same host, same minute, same randomized query parameters.
How this was found: a certified climb, not a hunch
We ran this as an optimization loop on the ai-functions
frontierbranch, with atif-sql main as the seeded incumbent. The loop only admits a candidate when hard gates hold and the objective beats the champion outside a measured noise band. The harness lives at~/bonk-fs/projects/atif-sql-frontier(sqlite archive plus event journal, resumable).atif-sqlbinary under GNU time. Gates before any ranking:uv sync --lockedinstalls,mise run checkis green, all tasks exit zero, the materializedtrajectory.jsonandedges.jsonlbytes match the frozen baseline, the three query outputs match the reference checkout run in the same evaluation on the same randomized date window and limit, the convert output matches, and peak RSS stays within 1.10x of baseline per task. Objective: geometric mean of per-task speedups.CertificationHarnessran known-bad candidates (a view that silently drops half the tool calls; a converter that skips side files) and degenerate candidates (empty, constant, and an uncached copy of the incumbent). The first two certification runs failed the degenerate probe: identical code measured 0.96 against frozen baseline medians because host conditions had drifted 4 percent in an hour, and short tasks jitter 3 to 8 percent on a single run. The fix was structural, not a wider bar: the reference checkout now runs interleaved with the candidate per repetition in alternating order, and the tie band is max(2 stderr, 5% / sqrt(reps)). Third run: all four probes passed. The known-bad candidates failed on exactly the gates their defects touch.9fd70b5What changed
Materialize runs its convert+write stage on a spawn
ProcessPoolExecutor(perf(corpus)9af371d).--workers NandATIF_SQL_MATERIALIZE_WORKERS, defaultmin(8, cpu_count);--workers 1is the untouched serial path. Each worker runs the same_write_session, so the per-session staging dir and atomic swap stay the crash-safety unit; outcomes fold back in plan order so the report is deterministic. Proof: 900 artifact files across 300 sessions, 0 differing between 1 and 8 workers; 51.2 s to 12.8 s on the Claude half alone.Cost estimation prices from litellm's bundled table without importing litellm (
perf(converter)04759f3). A profile of the 5.2 s convert showed 4.0 s wasimport litellm(openai, anthropic, hundreds of pydantic model builds) for onecost_per_tokencall per step.domain/pricing.pyreproduces litellm 1.100.1'sgeneric_cost_per_tokenarithmetic in the same operation order (text tokens = prompt minus cache read minus cache creation, service-tier suffixed rates, above-N-k tiers,ModelInfoBasefield filter, provider inference andget_model_inforesolution ladder) and readsmodel_prices_and_context_window_backup.jsonviafind_specso the package never imports. Anything it does not cover falls back to litellm exactly as before. Proof: 27,720 grid comparisons equal to the float, 0 divergences in the live parity oracle over both corpora, 0 of 378 trajectories differing by sha256 against main's binary;python -X importtimeshows no litellm line on the fast path. Convert 5.78 s to 2.89 s.Typed columnar artifacts so queries parse no JSON (
feat(duck)ed92aed). Materialize writessession.parquet,steps.parquet,tool_calls.parquet,tool_results.parquetbeside the four contract artifacts, inside the same staging dir and swap, claimed bymeta.columnar_schema = 1. atif-corpus gains anArtifactProducerport; atif-duck implements it with the same projection expressions the views use, so DuckDB normalizes the JSON columns identically; atif-cli composes them (--columnardefault on,--no-columnarfor exactly the four JSON artifacts). The registry readsread_parquetfor sessions that carry the artifacts and falls back toread_jsonper session otherwise, so older corpora keep working andatif-sql statusreportscolumnar|json|mixed. Proof:DESCRIBEequality plusEXCEPT ALLin both directions over every catalog view on JSON vs columnar vs mixed fixtures; byte-identical--format jsonfor five statements against main's binary on the same corpus; two tests destroy everytrajectory.jsonand require the views to keep answering, so a silent JSON fallback fails. Cost: +455 MB on disk (28 percent), and the write adds to materialize (single-process it was +74 percent wall), which is why this landed stacked on the pool.The merge commit threads the
ArtifactProducerthrough the pool initializer and addsartifact_secondsto each worker outcome.Things to know
--workers 1restores the old footprint.tasks_state_current's creation-order fallback now orders ties by(created_at, step_id, tool_use_id). Several TaskCreate calls in one step share a timestamp; the old window numbered them by physical scan order, which differed between the JSON and parquet readers and already collided on main. Panel outputs were unaffected.USE_TMP_FILE falseoverwrite hole for them (tested).LITELLM_LOCAL_MODEL_COST_MAPis unset, pricing now comes from the bundled table rather than a GitHub fetch at import; set it to any value other thantrueto route every call through litellm as before.mise run checkgreen at every commit (1329 passed, 1 skipped at the head);mise run docs:gategreen.