Skip to content

feat: Codex CLI transcript support end to end - #5

Merged
laithalsaadoon merged 2 commits into
mainfrom
feature/codex-transcripts
Sep 11, 2026
Merged

laithalsaadoon merged 2 commits into
mainfrom
feature/codex-transcripts

Conversation

@laithalsaadoon

@laithalsaadoon laithalsaadoon commented Sep 11, 2026

Copy link
Copy Markdown
Owner

atif-sql now reads Codex CLI rollouts alongside Claude Code sessions. One flag picks the agent — --agent claude-code|codex on convert, materialize, status and query — and it moves the default source and corpus roots with it, so the two agents never share a corpus directory.

What changed

atif-converter. A Codex adapter over harbor's private Codex._convert_events_to_trajectory, behind the same startup assertion the Claude Code path uses. Each rollout is staged ALONE into its own temp dir, because harbor folds every rollout it can see in a directory into one trajectory. CodexFidelityGap carries seven gaps, all values namespaced codex_* so one gaps_observed array serves either agent. Codex enrichment attributes agent steps by a re-derived api_call_id rather than by message text: harbor drops empty text parts, so an empty assistant message can never be placed by matching, which is what made 5 of 73 real rollouts refuse before the rewrite.

atif-corpus. Discovery is one value object per agent — transcript_depth 1 for Claude Code's <project>/<session>.jsonl, 3 for Codex's <YYYY>/<MM>/<DD> nesting — so the scanner walks either layout without branching on the agent, and an unlistable directory at any level still refuses to ghost the sessions under it. meta.json gains an agent key (absent in an older corpus reads as NULL and stays valid), and codex is reserved as an unhashed corpus slug. The agent enum is an AST-pinned twin of the converter's, because the independence contract forbids the import.

atif-duck. The sessions view gains agent and agent_version, and coalesces the two shapes harbor emits for working directory and git branch. steps coalesces the Codex cache-write token spelling into cache_creation.

Cron. The materialize lane runs one extra pass with --agent codex, guarded on the rollout root existing, on the resolved CLI actually carrying --agent, and on ATIF_SQL_CORPUS_ROOT not being pinned to the Claude corpus (one corpus holds one agent). The selftest proves each guard fires and stands down. Embedding does not ride the Codex pass: atif-sql embed takes no --agent, and threading the Codex Lance store through it is a separate change.

Also fixed. A missing transcript raised FileNotFoundError instead of a domain error on BOTH agents' paths, surfacing as an exit code indistinguishable from a crash.

Verified

mise run check green (960 tests), mise run docs:gate green, and scripts/atif-sql-refresh-selftest.sh reports 0 failures across 28 assertions.

Live on this host's 78 real rollouts: 77 materialized in 4.7 s (1 still live under quiescence), 1060 steps with every one enriched, all seven Codex gap members observed in the corpus, developer records preserved in edges.jsonl even though harbor flattens the role to system in the trajectory, and cost and token totals reported per Codex version.

Guards added after a pre-push review

A pre-push adversarial review of the first commit found five defects, four of them on the paths an unattended lane takes. All five are fixed in the second commit, and each guard was verified to fail when its fix is removed.

  • A pass aimed at the other agent's corpus deleted every session in it as a ghost, reported as "source vanished", exit 0. Per-agent default roots keep the agents apart, but an explicit --corpus-root or ATIF_SQL_CORPUS_ROOT defeats them. meta.agent is now a discriminator, read before ghost removal and before any write, and a disagreement exits 78 with nothing removed. A meta.json with no agent key answers claude-code, which is what protects the corpora already on disk.
  • A moved harbor private method became one failure per session under exit 0, so the cron lane logged materialize ok every ten minutes while nothing converted. The probe now runs once when the converter is built, and materialize maps it to 127.
  • The freshness dump skipped the Codex corpus, the one corpus with no lane of its own to report it.
  • enrichment_leftover_messages was documented and never written: running out of records refuses a step loudly, running out of steps was silent while the loss report still counted those records convertible.
  • convert ignored ATIF_SQL_AGENT, so the setting that moves the other three commands silently ran the Claude Code converter over a rollout and blamed the transcript.

Live check on the guard: a Codex pass pointed at a copy of a real Claude Code session exits 78 and leaves it alone. The Codex corpus still materializes all 78 rollouts. mise run check is green at 971 tests, mise run docs:gate is green, and the refresh selftest reports 0 failures across 30 assertions.

atif-sql now reads Codex CLI rollouts alongside Claude Code sessions. One
flag picks the agent — `--agent claude-code|codex` on convert, materialize,
status and query — and it moves the default source and corpus roots with it,
so a Codex corpus and a Claude Code corpus never share a directory.

Converter: `Codex._convert_events_to_trajectory` behind its own adapter with
a startup assertion, staging exactly one rollout per temp dir because harbor
folds a directory's rollouts into a single trajectory. `CodexFidelityGap`
carries the seven empirically observed gaps, values namespaced `codex_*` so
one `gaps_observed` array serves both agents. Codex enrichment attributes
agent steps by a re-derived `api_call_id` rather than by message text: harbor
drops empty text parts, so an empty assistant message can never be placed by
matching (5 of 73 real rollouts refused before this).

Corpus: per-agent `SourceLayout` (transcript depth 1 for Claude Code, 3 for
Codex's <YYYY>/<MM>/<DD> nesting), an AST-pinned twin of the converter's
AgentSource enum, `meta.agent` stamped per session, and `codex` reserved as
an unhashed corpus slug.

Duck: the sessions view gains `agent` and `agent_version`, and coalesces the
two shapes harbor emits for cwd and git branch; steps coalesces the Codex
cache-write token spelling.

Cron: the materialize lane runs one extra pass with `--agent codex`, guarded
on the rollout root existing, on the resolved CLI carrying `--agent`, and on
ATIF_SQL_CORPUS_ROOT not being pinned to the Claude corpus. The selftest
proves each guard fires and stands down.

Verified live on 78 rollouts: 77 materialized (1 still live), 1060 steps all
enriched, all seven gap members observed, and a missing transcript now raises
a domain error instead of leaking FileNotFoundError on both agents' paths.
RESPONSE_ITEM = "response_item"
EVENT_MSG = "event_msg"
TURN_CONTEXT = "turn_context"
TOKEN_USAGE_RECORD = "token_usage_record" # noqa: S105 — a record type, not a secret
Five defects, four of them in the paths an unattended lane takes.

1. A pass aimed at the other agent's corpus deleted every session in it as a
   ghost, reported as "source vanished", exit 0. Per-agent default roots keep
   the agents apart, but an explicit --corpus-root or ATIF_SQL_CORPUS_ROOT
   defeats them, and the README advertises that override. meta.agent is now a
   discriminator: materialize reads it before ghost removal and before any
   write, and a disagreement exits 78 with nothing removed. A meta.json with no
   agent key answers claude-code, because no corpus predating Codex support can
   hold Codex sessions — which is what protects the corpora already on disk.

2. A moved harbor private method became one failure per session under exit 0,
   so the cron lane logged "materialize ok" every ten minutes while nothing
   converted. The probe now runs once when RealConverter is built, and
   materialize maps it to 127, before the pass starts.

3. The freshness dump covered every corpus except the Codex one, which has no
   lane of its own to report it. It now reports there too, behind the same
   guards. A permanent "codex not yet supported" skip is documented as an
   operator signal (an older installed CLI wins resolution), not a steady state.

4. enrichment_leftover_messages was documented as the detector for message
   records that reach no step and was never written: running out of records
   refuses a step loudly, running out of steps was silent while the loss report
   still counted those records convertible. It is now written and logged.

5. convert ignored ATIF_SQL_AGENT, so the setting that moves materialize,
   status and query silently ran the Claude Code converter over a rollout and
   blamed the transcript ("no convertible events").

Each guard was verified to FAIL when its fix is removed: three corpus tests,
three CLI tests, two enrichment tests, and two shell selftest assertions.
Live: a Codex pass at a copy of a real Claude session exits 78 and leaves it
alone; the Codex corpus still materializes 78/78.
@laithalsaadoon
laithalsaadoon merged commit 99492d9 into main Sep 11, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants