feat: Codex CLI transcript support end to end - #5
Merged
Merged
Conversation
atif-sql now reads Codex CLI rollouts alongside Claude Code sessions. One flag picks the agent — `--agent claude-code|codex` on convert, materialize, status and query — and it moves the default source and corpus roots with it, so a Codex corpus and a Claude Code corpus never share a directory. Converter: `Codex._convert_events_to_trajectory` behind its own adapter with a startup assertion, staging exactly one rollout per temp dir because harbor folds a directory's rollouts into a single trajectory. `CodexFidelityGap` carries the seven empirically observed gaps, values namespaced `codex_*` so one `gaps_observed` array serves both agents. Codex enrichment attributes agent steps by a re-derived `api_call_id` rather than by message text: harbor drops empty text parts, so an empty assistant message can never be placed by matching (5 of 73 real rollouts refused before this). Corpus: per-agent `SourceLayout` (transcript depth 1 for Claude Code, 3 for Codex's <YYYY>/<MM>/<DD> nesting), an AST-pinned twin of the converter's AgentSource enum, `meta.agent` stamped per session, and `codex` reserved as an unhashed corpus slug. Duck: the sessions view gains `agent` and `agent_version`, and coalesces the two shapes harbor emits for cwd and git branch; steps coalesces the Codex cache-write token spelling. Cron: the materialize lane runs one extra pass with `--agent codex`, guarded on the rollout root existing, on the resolved CLI carrying `--agent`, and on ATIF_SQL_CORPUS_ROOT not being pinned to the Claude corpus. The selftest proves each guard fires and stands down. Verified live on 78 rollouts: 77 materialized (1 still live), 1060 steps all enriched, all seven gap members observed, and a missing transcript now raises a domain error instead of leaking FileNotFoundError on both agents' paths.
Five defects, four of them in the paths an unattended lane takes.
1. A pass aimed at the other agent's corpus deleted every session in it as a
ghost, reported as "source vanished", exit 0. Per-agent default roots keep
the agents apart, but an explicit --corpus-root or ATIF_SQL_CORPUS_ROOT
defeats them, and the README advertises that override. meta.agent is now a
discriminator: materialize reads it before ghost removal and before any
write, and a disagreement exits 78 with nothing removed. A meta.json with no
agent key answers claude-code, because no corpus predating Codex support can
hold Codex sessions — which is what protects the corpora already on disk.
2. A moved harbor private method became one failure per session under exit 0,
so the cron lane logged "materialize ok" every ten minutes while nothing
converted. The probe now runs once when RealConverter is built, and
materialize maps it to 127, before the pass starts.
3. The freshness dump covered every corpus except the Codex one, which has no
lane of its own to report it. It now reports there too, behind the same
guards. A permanent "codex not yet supported" skip is documented as an
operator signal (an older installed CLI wins resolution), not a steady state.
4. enrichment_leftover_messages was documented as the detector for message
records that reach no step and was never written: running out of records
refuses a step loudly, running out of steps was silent while the loss report
still counted those records convertible. It is now written and logged.
5. convert ignored ATIF_SQL_AGENT, so the setting that moves materialize,
status and query silently ran the Claude Code converter over a rollout and
blamed the transcript ("no convertible events").
Each guard was verified to FAIL when its fix is removed: three corpus tests,
three CLI tests, two enrichment tests, and two shell selftest assertions.
Live: a Codex pass at a copy of a real Claude session exits 78 and leaves it
alone; the Codex corpus still materializes 78/78.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
atif-sqlnow reads Codex CLI rollouts alongside Claude Code sessions. One flag picks the agent —--agent claude-code|codexonconvert,materialize,statusandquery— and it moves the default source and corpus roots with it, so the two agents never share a corpus directory.What changed
atif-converter. A Codex adapter over harbor's private
Codex._convert_events_to_trajectory, behind the same startup assertion the Claude Code path uses. Each rollout is staged ALONE into its own temp dir, because harbor folds every rollout it can see in a directory into one trajectory.CodexFidelityGapcarries seven gaps, all values namespacedcodex_*so onegaps_observedarray serves either agent. Codex enrichment attributes agent steps by a re-derivedapi_call_idrather than by message text: harbor drops empty text parts, so an empty assistant message can never be placed by matching, which is what made 5 of 73 real rollouts refuse before the rewrite.atif-corpus. Discovery is one value object per agent —
transcript_depth1 for Claude Code's<project>/<session>.jsonl, 3 for Codex's<YYYY>/<MM>/<DD>nesting — so the scanner walks either layout without branching on the agent, and an unlistable directory at any level still refuses to ghost the sessions under it.meta.jsongains anagentkey (absent in an older corpus reads as NULL and stays valid), andcodexis reserved as an unhashed corpus slug. The agent enum is an AST-pinned twin of the converter's, because the independence contract forbids the import.atif-duck. The
sessionsview gainsagentandagent_version, and coalesces the two shapes harbor emits for working directory and git branch.stepscoalesces the Codex cache-write token spelling intocache_creation.Cron. The materialize lane runs one extra pass with
--agent codex, guarded on the rollout root existing, on the resolved CLI actually carrying--agent, and onATIF_SQL_CORPUS_ROOTnot being pinned to the Claude corpus (one corpus holds one agent). The selftest proves each guard fires and stands down. Embedding does not ride the Codex pass:atif-sql embedtakes no--agent, and threading the Codex Lance store through it is a separate change.Also fixed. A missing transcript raised
FileNotFoundErrorinstead of a domain error on BOTH agents' paths, surfacing as an exit code indistinguishable from a crash.Verified
mise run checkgreen (960 tests),mise run docs:gategreen, andscripts/atif-sql-refresh-selftest.shreports 0 failures across 28 assertions.Live on this host's 78 real rollouts: 77 materialized in 4.7 s (1 still live under quiescence), 1060 steps with every one enriched, all seven Codex gap members observed in the corpus,
developerrecords preserved inedges.jsonleven though harbor flattens the role tosystemin the trajectory, and cost and token totals reported per Codex version.Guards added after a pre-push review
A pre-push adversarial review of the first commit found five defects, four of them on the paths an unattended lane takes. All five are fixed in the second commit, and each guard was verified to fail when its fix is removed.
--corpus-rootorATIF_SQL_CORPUS_ROOTdefeats them.meta.agentis now a discriminator, read before ghost removal and before any write, and a disagreement exits 78 with nothing removed. Ameta.jsonwith noagentkey answersclaude-code, which is what protects the corpora already on disk.materialize okevery ten minutes while nothing converted. The probe now runs once when the converter is built, andmaterializemaps it to 127.enrichment_leftover_messageswas documented and never written: running out of records refuses a step loudly, running out of steps was silent while the loss report still counted those records convertible.convertignoredATIF_SQL_AGENT, so the setting that moves the other three commands silently ran the Claude Code converter over a rollout and blamed the transcript.Live check on the guard: a Codex pass pointed at a copy of a real Claude Code session exits 78 and leaves it alone. The Codex corpus still materializes all 78 rollouts.
mise run checkis green at 971 tests,mise run docs:gateis green, and the refresh selftest reports 0 failures across 30 assertions.