Add Grok CLI harness support

[willwashburn]

Summary

Add Grok CLI as a fourth harness in burn with maximum feature parity to Claude Code, Codex, and OpenCode. Grok sessions live under ~/.grok/sessions/<url-encoded-cwd>/<session-id>/ but do not currently emit per-turn billing tokens in on-disk logs, so usage will be estimated until native usage fields appear in session files.

Motivation

Grok CLI persists rich session data locally (chat_history.jsonl, updates.jsonl, summary.json, prompt_context.json, signals.json) but burn does not ingest it today. Users running Grok alongside other harnesses cannot get unified burn summary, hotspots, compare, or overhead views.

Feature parity matrix

Capability	Claude	Codex	Grok target	Notes
burn ingest / --watch	Yes	Yes	Yes	Scan ~/.grok/sessions/
burn summary	Yes	Yes	Yes	Estimated tokens + priced cost
burn hotspots	Yes	Yes	Yes	Tool/file/bash attribution from chat_history
burn compare	Yes	Yes	Yes	Activity classification
burn overhead	CLAUDE.md	AGENTS.md	Yes	prompt_context.json + Agents.md
Pending stamps	Yes	Yes	Yes	writePendingStamp({ harness: "grok" })
Harness adapter	Yes	Yes	Yes	pending_stamp + watch loop
Subagent tree	Yes	Yes	Partial → Yes	Task tool + subagents/ child sessions
Compaction events	Yes	Yes	Partial	compaction_checkpoints/, signals.compactionCount
Per-turn billing fidelity	Full	Full	Partial	Estimate; mark FidelityClass::Partial
Cache attribution	Yes	Partial	No	Not in Grok logs
Provider grouping	Yes	Yes	Yes	xai

Grok session layout

~/.grok/sessions/<url-encoded-cwd>/<session-id>/
  chat_history.jsonl   ← primary: turns, tools, content (API message transcript)
  updates.jsonl        ← secondary: turnStartMs, totalTokens, timestamps (ACP/UI stream)
  summary.json         ← session id, cwd, model, git metadata
  prompt_context.json  ← AGENTS.md snapshot for overhead
  signals.json         ← session aggregates (sanity check)
  subagents/           ← child session metadata
  compaction_checkpoints/

Design choice: parse chat_history.jsonl for semantic content; join updates.jsonl for turn boundaries and context-size proxies.

chat_history.jsonl record types: system, user, reasoning, assistant, tool_result.

Unlike Claude/Codex (single authoritative JSONL with billing usage blocks), Grok splits model transcript from UI stream and does not log input_tokens / output_tokens / cache breakdown per turn.

Pricing

Known Grok rates (to ship as xAI model overrides):

Metric	Cost
Input tokens	$1.00 / 1M
Output tokens	$2.00 / 1M

Model IDs observed in sessions:

• grok-composer-2.5-fast
• grok-build

Add to vendored models.dev.json and support $RELAYBURN_HOME/models.dev.json overrides. No cache pricing (not exposed in logs).

Usage estimation strategy

For each turn (bounded by turnStartMs changes in updates.jsonl, aligned with <user_query> → assistant completion in chat_history):

1. Output tokens — HeuristicCounter (bytes/4) over reasoning.summary, assistant.content, and serialized tool_calls[].arguments.
2. Input tokens — prefer context proxy when available: input ≈ max(0, totalTokens_end − totalTokens_start) for that turnStartMs, subtract estimated output, floor at user-prompt heuristic size. Fallback: full heuristic on messages since prior turn.
3. Reasoning tokens — count reasoning summary text separately.
4. Fidelity — FidelityClass::Partial, UsageGranularity::PerTurn; coverage: input/output true, cache false.
5. Validation — cross-check against signals.json (contextTokensUsed, turnCount); warn if drift >15%.

burn summary should surface a fidelity note when Grok turns are present (similar to missing-pricing warnings).

Future: if Grok adds usage blocks to session logs, feature-detect and parse natively (Claude-style) behind the estimator.

Implementation plan (PR stack)

PR 1 — Types & enums (relayburn-sdk)

• SourceKind::Grok ("grok")
• RelationshipSourceKind::Grok, NativeGrok
• PendingStampHarness::Grok
• IngestRoots.grok_sessions_dir (default ~/.grok/sessions)
• AdapterName::Grok, FileCursor::Grok(GrokCursor)
• Node SDK + index.d.ts updates

Files: reader/types.rs, ingest/cursors.rs, ingest/gap.rs, ingest/ingest.rs, pending_stamps.rs, relayburn-sdk-node, packages/sdk-node

PR 2 — Grok reader (reader/grok.rs)

parse_grok_session_incremental(session_dir, opts) -> ParseGrokIncrementalResult

• Walk chat_history.jsonl; start turn on user with <user_query>
• Accumulate reasoning, assistant, tool_result until next user query
• Metadata from summary.json
• Map tool_calls / tool_result to burn content model
• Incremental GrokCursor (chat_history + updates offsets)
• Fixtures in tests/fixtures/grok/

PR 3 — Ingest orchestration

• ingest_grok_into(), ingest_grok_sessions()
• Wire into ingest_all(), default_session_roots(), source_fingerprint()
• Pending-stamp resolution, gap warning adapter

PR 4 — Classifier & tool aliases

Grok tool	Canonical
Shell	Bash
Read	Read
Write	Write
StrReplace / Edit	Edit
Grep	Grep
Glob	Glob
Task	Task
WebSearch / WebFetch	WebFetch
CallMcpTool	Mcp

PR 5 — Pricing

Add xAI entries to models.dev.json for grok-composer-2.5-fast, grok-build, and fallback aliases.

PR 6 — Harness adapter & registry

• crates/relayburn-cli/src/harnesses/grok.rs via pending_stamp::session_store_adapter
• Register in registry.rs; update harness name tests

PR 7 — Overhead (AGENTS.md)

• Treat SourceKind::Grok like Codex/OpenCode for AGENTS.md
• Optionally ingest prompt_context.json for overhead attribution

PR 8 — Subagents & relationships

• Parse Task tool calls; walk subagents/ for child sessions
• Emit SessionRelationshipRecord with NativeGrok
• Update subagent_tree tests

PR 9 — Node SDK + MCP

• PendingStampHarness::Grok, ingest harness option, overhead harness
• MCP fixture test with Grok-ingested session

PR 10 — Docs & changelog

• README.md, Agents.md, CHANGELOG.md, packages/sdk-node/CHANGELOG.md
• Document partial-fidelity caveat

PR 11 — Integration tests

• SDK integration: ingest fixture → summary with non-zero turns
• CLI smoke: pinned grok_sessions_dir
• Fidelity: grok turns classified partial

Suggested milestones

MVP (PRs 1–5, 3, 10): ingest + summary + hotspots + pricing. Partial fidelity; no subagents.

Follow-up (PRs 6–8): harness adapter, overhead, subagents.

Known gaps (document in README)

1. No native billing tokens — costs are estimates
2. No cache read/create attribution
3. totalTokens can decrease on compaction — input math must handle resets
4. Encrypted reasoning blobs excluded from token counts (only summary text)
5. Model ID drift — alias table may need updates

Verification

cargo test --workspace
cargo run -p relayburn-cli -- ingest
cargo run -p relayburn-cli -- summary --since 7d
cargo run -p relayburn-cli -- hotspots --project .
cargo run -p relayburn-cli -- overhead --kind agents-md
cargo run -p relayburn-cli -- compare --since 30d
pnpm run test

Manual: run a short grok session, then confirm burn summary shows grok-composer-* with estimated cost.

References

• Grok session docs: ~/.grok/README.md (Session Persistence section)
• Grok storage: ~/.grok/sessions/
• Burn harness pattern: Agents.md → "Adding a harness"
• Codex reader reference: crates/relayburn-sdk/src/reader/codex.rs
• OpenCode ingest reference: crates/relayburn-cli/src/harnesses/opencode.rs