Skip to content

feat(config): make the codex runtime selectable - #6927

Merged
waynesun09 merged 11 commits into
mainfrom
codex-runtime-enable-docs
Sep 3, 2026
Merged

feat(config): make the codex runtime selectable#6927
waynesun09 merged 11 commits into
mainfrom
codex-runtime-enable-docs

Conversation

@waynesun09

@waynesun09 waynesun09 commented Sep 2, 2026

Copy link
Copy Markdown
Member

Summary

Makes runtime: codex selectable and documents it.

  • config.ValidRuntimes() += codex and every enumeration (interactive setup prompt, --runtime help on run/agent/github/repos, stale-image warning for CODEX_VERSION, CLI reference tables, architecture/glossary/getting-started guides).
  • FULLSEND_CODEX_MODEL: a runtime-scoped model override generalised from the pi-only case (runtimeModelEnv), gated on the selected runtime, echoed as its own source in the plan block and override_source; forwarded into the agent job by setup-agent-env.sh (with shell tests). Codex requires an explicit OpenAI model, and fleet harnesses say model: opus, so this is the per-repo default route; model: openai/<id> on the agents: entry is the other.
  • docs/runtimes/codex.md as a walkthrough (example → models → at a glance → running it locally → behaviour differences → not yet exercised → troubleshooting) with error and guard texts quoted verbatim from PR D; runtimes.md status/comparison/config-key/precedence rows; local-run and WIF guides; VitePress sidebar.
  • Gated behaviour scenario features/runtime/codex-openai.feature (runtime-codex-openai, not in the default capability set — codex has no Vertex path, so it has no default behaviour-CI coverage until an OpenAI org WIF mapping exists) with a structural output-stream step.

Review rounds

sol + Grok, then a Grok re-review: CRITICAL (FULLSEND_CODEX_MODEL missing from the workflow's variable allowlist) fixed with tests; four doc claims corrected against PR D's code (identity effort mapping, tools: recorded not enforced, canary block vs sanitizer warn, spawn_agent exists but no roster); every remaining pi-only enumeration and copy site updated.

Verification (macOS, image from PR A)

Fleet-shaped triage and review from the agents clone on codex: fleet model: opus refused with the documented text; FULLSEND_CODEX_MODEL=openai/gpt-5.6-luna → triage exit 0 with labels and comment, review exit 0 with risk/low applied (post-script 422 only because a PR cannot be reviewed by its own author). Whole stack builds and tests green on the Fedora x86_64 host.


Part of a five-PR stack for #6920 (Codex as an agent runtime): A image pin → B stream parser → C OpenAI credential seeder → D runtime core (ADR 0099) → E enable + docs. Each PR is reviewable on its own diff; they merge bottom-up. Plan and verified Codex facts: research/fullsend-codex-runtime-plan.md in the ai-workspace-public research repo (to be linked once pushed).

Refs #6920

Assisted-by: Claude (implementation and review orchestration), Codex gpt-5.6-sol (review), Grok 4.6 (review)

Pre-existing bug fixed here

The six standalone reusable role workflows (reusable-{triage,review,code,fix,prioritize,retro}.yml) called setup-agent-env.sh without FULLSEND_REPO_VARS, so no repository override (FULLSEND_MODEL, FULLSEND_EFFORT, FULLSEND_RUNTIME, pi's knobs) reached runs on that route — only reusable-dispatch.yml passed them. Codex made it visible because a dropped override there is a refused run rather than a silently wrong model. Fixed by passing the variables on all six, mirroring dispatch.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Enable Codex runtime selection and model overrides

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Makes Codex selectable across repository, agent, setup, and fleet configuration.
• Adds runtime-scoped Codex model overrides with workflow forwarding and precedence tests.
• Documents Codex operations and adds gated OpenAI WIF behavior coverage.
Diagram

graph TD
  CFG["Runtime config"] --> RES["Override resolver"] --> REG["Runtime registry"] --> CX["Codex runtime"] --> OA["OpenAI API"]
  ENV["Model variables"] --> RES
  CX --> ART["Run artifacts"]
  TEST["Behaviour tests"] --> CX
Loading
High-Level Assessment

The approach is appropriate for enabling an already-implemented runtime: add Codex to the centralized valid-runtime list, generalize the existing pi-specific model alias mechanism, and gate live WIF coverage until suitable infrastructure exists. Using only FULLSEND_MODEL was considered, but FULLSEND_CODEX_MODEL provides the needed repository-wide default without overriding models for other runtimes or editing fleet harnesses.

Files changed (44) +876 / -145

Enhancement (11) +63 / -39
agent.goExpose Codex in agent runtime help +1/-1

Expose Codex in agent runtime help

• Adds Codex to the supported values shown by the agent runtime flag.

internal/cli/agent.go

github.goEnable Codex setup messaging and warnings +5/-2

Enable Codex setup messaging and warnings

• Adds Codex to GitHub setup flag help and OpenAI WIF descriptions. Warns when selected repositories may use stale images without CODEX_VERSION.

internal/cli/github.go

repos.goExpose Codex in repository fleet help +1/-1

Expose Codex in repository fleet help

• Adds Codex to the runtime values accepted and described by repository installation commands.

internal/cli/repos.go

run.goIntegrate Codex into run override metadata +14/-12

Integrate Codex into run override metadata

• Adds Codex to runtime/model CLI help and records FULLSEND_CODEX_MODEL as a possible model source. Generalizes comments around the two-pass runtime-scoped override resolution.

internal/cli/run.go

run_overrides.goGeneralize runtime-scoped model overrides +27/-3

Generalize runtime-scoped model overrides

• Introduces FULLSEND_CODEX_MODEL and maps runtime names to their scoped model environment variables. Preserves flag and FULLSEND_MODEL precedence while reading aliases only for the selected runtime.

internal/cli/run_overrides.go

runtime_prompt.goOffer Codex in the interactive runtime prompt +2/-1

Offer Codex in the interactive runtime prompt

• Adds an experimental Codex choice and updates pilot guidance while retaining Claude Code as the recommended default.

internal/cli/runtime_prompt.go

config.goMake Codex a valid configured runtime +5/-4

Make Codex a valid configured runtime

• Adds Codex to ValidRuntimes so repository, per-agent, setup, and manifest validation can select it.

internal/config/config.go

manifest.goDocument Codex manifest runtime values +3/-3

Document Codex manifest runtime values

• Updates repository entry and fleet default field comments to include Codex.

internal/repos/manifest.go

scaffold_metadata.goAdd Codex to scaffold runtime instructions +1/-1

Add Codex to scaffold runtime instructions

• Updates generated setup metadata so users can switch repositories to Claude, pi, or Codex.

internal/repos/scaffold_metadata.go

codex.goRemove Codex selection-gate comments +1/-6

Remove Codex selection-gate comments

• Updates the runtime declaration to reflect that Codex is now selectable through repository and agent configuration.

internal/runtime/codex.go

registry.goPromote Codex from stub to selectable backend +3/-5

Promote Codex from stub to selectable backend

• Removes stale stub annotations and leaves only OpenCode excluded from user-facing configuration.

internal/runtime/registry.go

Tests (9) +401 / -32
codex-openai.featureAdd gated Codex OpenAI WIF scenario +77/-0

Add gated Codex OpenAI WIF scenario

• Adds an opt-in behavior scenario proving Codex selection, WIF-backed execution, completed tool use, and token metrics. The scenario remains disabled without the runtime-codex-openai capability.

e2e/behaviour/features/runtime/codex-openai.feature

run_overrides_test.goTest Codex model override precedence +61/-0

Test Codex model override precedence

• Covers runtime-to-variable mapping, Codex alias gating, cross-runtime isolation, and precedence against flags and FULLSEND_MODEL.

internal/cli/run_overrides_test.go

runtime_prompt_test.goTest interactive Codex selection +8/-1

Test interactive Codex selection

• Verifies Codex input and ensures every accepted human runtime is visibly listed in the prompt.

internal/cli/runtime_prompt_test.go

config_test.goValidate Codex configuration acceptance +9/-9

Validate Codex configuration acceptance

• Updates valid-runtime, organization defaults, and per-repository tests to accept Codex while continuing to reject OpenCode.

internal/config/config_test.go

registry_test.goTest configured Codex backend resolution +23/-7

Test configured Codex backend resolution

• Verifies repository-wide and per-agent Codex resolution and removes Codex from rejected-stub test cases.

internal/runtime/registry_test.go

setup-agent-env-test.shTest Codex workflow variable forwarding +12/-0

Test Codex workflow variable forwarding

• Verifies plain and role-prefixed FULLSEND_CODEX_MODEL repository variables are forwarded to agent jobs with the expected precedence.

internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env-test.sh

scaffold_test.goVerify Codex variable exists in workflow scaffold +1/-1

Verify Codex variable exists in workflow scaffold

• Extends scaffold content assertions to require FULLSEND_CODEX_MODEL passthrough support.

internal/scaffold/scaffold_test.go

runtime.goAdd Codex behavior-test steps and stream parsing +150/-9

Add Codex behavior-test steps and stream parsing

• Generalizes runtime-agent fixture creation and adds Codex-specific JSONL artifact detection. The new assertion requires a successfully completed command_execution event rather than announced, declined, or nested events.

pkg/behaviourtest/steps/runtime.go

runtime_test.goTest Codex behavior artifact assertions +60/-5

Test Codex behavior artifact assertions

• Covers shared agent fixture setup and Codex stream classification, completed command detection, declined commands, wrong artifact paths, nested markers, and malformed input.

pkg/behaviourtest/steps/runtime_test.go

Documentation (22) +407 / -73
config.tsAdd Codex runtime to documentation navigation +1/-0

Add Codex runtime to documentation navigation

• Adds the Codex runtime guide to the VitePress runtime sidebar.

docs/.vitepress/config.ts

architecture.mdRepresent Codex in the runtime architecture +9/-4

Represent Codex in the runtime architecture

• Adds Codex, its sandbox configuration, hook integration, and OpenAI data path to the architecture diagram and runtime inventory. Removes language describing Codex as non-selectable.

docs/architecture.md

README.mdExpand inference credentials summary for Codex +1/-1

Expand inference credentials summary for Codex

• Clarifies that OpenAI WIF enrollment supports GPT workloads on both pi and Codex.

docs/cli/README.md

agent.mdDocument Codex agent settings +2/-2

Document Codex agent settings

• Adds Codex to the agent runtime flag and explains its OpenAI-only model references.

docs/cli/agent.md

github.mdDocument Codex GitHub setup flags +2/-2

Document Codex GitHub setup flags

• Adds Codex to setup runtime choices and extends OpenAI WIF flag descriptions to cover Codex.

docs/cli/github.md

repos.mdDocument Codex fleet runtime configuration +2/-2

Document Codex fleet runtime configuration

• Adds Codex to repository installation flags and the supported defaults.runtime values.

docs/cli/repos.md

run.mdDocument Codex run overrides and artifacts +8/-8

Document Codex run overrides and artifacts

• Adds Codex to runtime/model flags, override precedence, metrics sources, debug artifacts, and OpenAI credential guidance.

docs/cli/run.md

runtime-implementation.mdMark Codex as selectable +1/-2

Mark Codex as selectable

• Updates runtime implementation guidance to reflect that Codex can now be selected while retaining its outstanding coverage limitations.

docs/contributing/runtime-implementation.md

sandbox-topology.mdClassify Codex as an opt-in image runtime +1/-1

Classify Codex as an opt-in image runtime

• Replaces the Codex stub designation with its opt-in runtime status in the sandbox image inventory.

docs/contributing/sandbox-topology.md

glossary.mdAdd Codex to the agent runtime definition +1/-1

Add Codex to the agent runtime definition

• Describes Codex alongside pi as an available opt-in runtime while leaving OpenCode classified as a stub.

docs/glossary.md

behaviour-testing.mdDocument Codex behavior-test conventions +1/-1

Document Codex behavior-test conventions

• Explains shared runtime-agent setup, Codex stream assertions, capability gating, and the lack of default live Codex coverage.

docs/guides/dev/behaviour-testing.md

cli-internals.mdExtend OpenAI enrollment internals to Codex +1/-1

Extend OpenAI enrollment internals to Codex

• Clarifies that the OpenAI inference command supports GPT on pi or Codex.

docs/guides/dev/cli-internals.md

choosing-a-runtime.mdAdd Codex to runtime selection guidance +3/-2

Add Codex to runtime selection guidance

• Introduces Codex as an experimental OpenAI-only option and documents its model, role, and setup constraints.

docs/guides/getting-started/choosing-a-runtime.md

configuring-github.mdExpose Codex during GitHub configuration +1/-1

Expose Codex during GitHub configuration

• Identifies Codex as an experimental, opt-in choice during repository setup.

docs/guides/getting-started/configuring-github.md

operations.mdInclude Codex in OpenAI operations +1/-1

Include Codex in OpenAI operations

• Extends the OpenAI WIF operations table to cover Codex workloads.

docs/guides/getting-started/operations.md

layered-config-reference.mdAdd Codex to layered configuration values +2/-2

Add Codex to layered configuration values

• Lists Codex as a valid runtime and notes that layered OpenAI WIF identifiers serve pi and Codex.

docs/guides/infrastructure/layered-config-reference.md

openai-workload-identity.mdDocument OpenAI WIF for Codex +20/-9

Document OpenAI WIF for Codex

• Extends enrollment, model configuration, local execution, and related guidance from pi-only usage to Codex. Documents Codex model restrictions and FULLSEND_CODEX_MODEL.

docs/guides/infrastructure/openai-workload-identity.md

running-agents-locally.mdAdd local Codex execution guidance +17/-7

Add local Codex execution guidance

• Explains how to run Codex locally with an OpenAI key and model while preserving the previous pi-only anchor for compatibility.

docs/guides/user/running-agents-locally.md

runtimes.mdIntegrate Codex into the runtime reference +37/-24

Integrate Codex into the runtime reference

• Adds Codex to status, comparison, precedence, model, harness-key, security, and artifact documentation. Reclassifies it from stub to selectable opt-in runtime.

docs/runtimes.md

codex.mdAdd comprehensive Codex runtime guide +294/-0

Add comprehensive Codex runtime guide

• Introduces Codex models, configuration, local execution, artifacts, security behavior, limitations, verification status, and troubleshooting. Documents FULLSEND_CODEX_MODEL and exact failure guidance.

docs/runtimes/codex.md

pi.mdUpdate shared OpenAI local-run link +1/-1

Update shared OpenAI local-run link

• Points pi users to the renamed local OpenAI credential section shared with Codex.

docs/runtimes/pi.md

README.mdMark the pinned Codex binary as active +1/-1

Mark the pinned Codex binary as active

• Updates the image component table to describe Codex as an opt-in runtime rather than a stub.

images/README.md

Other (2) +5 / -1
MakefileDocument gated Codex behavior capability +4/-0

Document gated Codex behavior capability

• Explains why Codex OpenAI behavior coverage is excluded from the default capability set and how to enable it once an OpenAI organization mapping exists.

Makefile

setup-agent-env.shForward Codex model variables into agent jobs +1/-1

Forward Codex model variables into agent jobs

• Adds FULLSEND_CODEX_MODEL to the workflow repository-variable allowlist used to populate the agent environment.

internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 6:54 PM UTC · Ended 6:57 PM UTC

Commit: 84e2214 · View workflow run →

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Site preview

Preview: https://25b6369e-site.fullsend-ai.workers.dev

Commit: 5bf423c57f3a03ccba6a6675ea68bba1b8ac0218

@qodo-code-review

qodo-code-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Codex model variable not forwarded ✓ Resolved 🐞 Bug ≡ Correctness
Description
Standalone reusable role workflows invoke setup-agent-env.sh without FULLSEND_REPO_VARS, so
repository overrides—including the newly allowlisted plain and role-prefixed Codex model
variables—are never exported for those agent runs. A Codex job therefore retains a fleet harness
model such as opus, which translateCodexModel rejects before execution.
Code

internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh[45]

+  override_keys=(FULLSEND_RUNTIME FULLSEND_MODEL FULLSEND_EFFORT FULLSEND_FALLBACK_MODELS FULLSEND_PI_PROVIDER FULLSEND_PI_MODEL FULLSEND_CODEX_MODEL)
Relevance

●●● Strong

Reusable workflows omit the variable source needed for Codex model overrides, causing valid CI
configurations to fail.

PR-#6874

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The setup script reads and exports allowlisted repository overrides only inside the
FULLSEND_REPO_VARS conditional. The direct dispatch workflow supplies toJSON(vars), but the
reusable code, triage, review, fix, prioritize, and retro workflows invoke the script without
defining that variable, even though these standalone workflows are included in the vendored workflow
manifest; consequently, the Codex override cannot reach resolveRunOverrides, and Codex rejects the
inherited Claude alias rather than translating or replacing it with an OpenAI model.

internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh[41-55]
.github/workflows/reusable-dispatch.yml[659-668]
.github/workflows/reusable-review.yml[163-169]
.github/workflows/reusable-triage.yml[155-161]
internal/scaffold/vendormanifest.go[137-145]
internal/runtime/codex_run.go[96-113]
internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh[42-55]
.github/workflows/reusable-code.yml[168-175]
internal/cli/run_overrides.go[100-108]
internal/runtime/codex_run.go[96-118]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description

`FULLSEND_CODEX_MODEL` is allowlisted by `setup-agent-env.sh`, but standalone reusable role workflows do not pass repository variables through `FULLSEND_REPO_VARS`. As a result, plain and role-prefixed Codex model overrides are unavailable, leaving Codex with fleet harness defaults such as `opus`, which it rejects before execution.

## Issue Context

Pass `FULLSEND_REPO_VARS: ${{ toJSON(vars) }}` to each reusable workflow's `Setup agent environment` step, matching the behavior already demonstrated by `reusable-dispatch.yml`. Ensure every reusable agent role receives consistent override forwarding, and add coverage that prevents the workflow variants from diverging.

## Fix Focus Areas

- internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh[42-55]
- .github/workflows/reusable-code.yml[168-175]
- .github/workflows/reusable-fix.yml[348-359]
- .github/workflows/reusable-prioritize.yml[137-144]
- .github/workflows/reusable-retro.yml[150-156]
- .github/workflows/reusable-review.yml[163-169]
- .github/workflows/reusable-triage.yml[155-161]
- internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env-test.sh[49-59]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Admin help omits codex ✗ Dismissed 📘 Rule violation ⚙ Maintainability
Description
The shared runtime enumeration now accepts codex through the per-repository `fullsend admin
install path, but that command's --runtime help still lists only claude, pi, and dummy`.
Users therefore receive stale CLI guidance for behavior activated by this PR.
Code

internal/config/config.go[312]

+	return []string{"claude", "pi", "codex", "dummy", "dummy-playback"}
Relevance

●●● Strong

Help text omits a newly valid runtime, a deterministic CLI documentation inconsistency.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2748504 requires user-facing CLI documentation to remain consistent with changed behavior.
config.ValidRuntimes() now includes codex, and the admin command validates against that list,
but its help string omits Codex.

Rule 2748504: Update docs when changing CLI behavior or public API
internal/config/config.go[311-312]
internal/cli/admin.go[630-630]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The legacy admin command's runtime help does not reflect the runtime values accepted by its per-repository path.

## Issue Context
Clarify that Codex is accepted for supported per-repository installation while remaining unavailable for deprecated per-org installation.

## Fix Focus Areas
- internal/config/config.go[311-312]
- internal/cli/admin.go[630-630]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Codex prompt misstates sub-agents ✓ Resolved 📘 Rule violation ⚙ Maintainability
Description
The new interactive prompt says Codex has no sub-agent tool, while the Codex documentation states
that spawn_agent exists but Fullsend does not build its persona roster. This gives users an
inaccurate explanation of the actual limitation.
Code

internal/cli/runtime_prompt.go[35]

+	printer.StepInfo("  [codex]  Codex — experimental; OpenAI models only, no sub-agent tool (review/retro stay on claude)")
Relevance

●●● Strong

Prompt contradicts the new Codex documentation, making this a clear user-facing accuracy fix.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2748504 requires user-facing messages and documentation to describe changed CLI behavior
consistently. The newly added prompt denies the existence of a sub-agent tool, whereas the new Codex
guide explicitly documents spawn_agent and identifies missing roster wiring as the limitation.

Rule 2748504: Update docs when changing CLI behavior or public API
internal/cli/runtime_prompt.go[35-35]
docs/runtimes/codex.md[191-193]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The runtime prompt incorrectly says Codex has no sub-agent tool.

## Issue Context
Codex exposes `spawn_agent`; Fullsend's current limitation is that it does not construct the persona roster required by `review` and `retro`. Match the concise prompt wording to the detailed runtime documentation.

## Fix Focus Areas
- internal/cli/runtime_prompt.go[35-35]
- docs/runtimes/codex.md[191-193]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. runtimeModelEnv has one caller ✗ Dismissed 📘 Rule violation ⌂ Architecture
Description
The new runtimeModelEnv helper is used by only one production call path, resolveRunOverrides.
With no second production consumer or consolidation of multiple existing implementations, the
abstraction should be inlined.
Code

internal/cli/run_overrides.go[70]

+func runtimeModelEnv(runtimeName string) string {
Relevance

●● Moderate

Single-use abstraction concerns are subjective; helper intentionally centralizes runtime-specific
aliases for current and future runtimes.

PR-#6360

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 1062065 requires at least two distinct production usage sites for a new abstraction unless it
consolidates multiple duplicate implementations. The helper is defined here and called only once
from resolveRunOverrides; its remaining references are tests and comments.

Rule 1062065: Avoid single-use abstractions; require reuse across multiple call paths
internal/cli/run_overrides.go[63-78]
internal/cli/run_overrides.go[100-107]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`runtimeModelEnv` introduces a helper abstraction with only one production caller.

## Issue Context
Tests do not count as distinct production usage under the compliance rule. Preserve the Pi and Codex mappings and precedence behavior while moving the mapping into the sole resolver call path.

## Fix Focus Areas
- internal/cli/run_overrides.go[63-78]
- internal/cli/run_overrides.go[100-107]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (1)
5. Codex extends deprecated org mode ✗ Dismissed 📘 Rule violation § Compliance
Description
Adding codex to config.ValidRuntimes() also makes it valid for deprecated per-org `fullsend
admin install` flows. This unintentionally extends deprecated functionality without the required ADR
0044 disclosure.
Code

internal/config/config.go[312]

+	return []string{"claude", "pi", "codex", "dummy", "dummy-playback"}
Relevance

●● Moderate

Deprecated org-mode scope is a specific compliance concern, but historical treatment of shared
enumerations is mixed.

PR-#2407

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2795055 prohibits extending deprecated per-org installation mode. The changed shared
enumeration includes codex, while the per-org admin path validates its selected runtime directly
against that enumeration, thereby activating Codex for org installs.

Rule 2795055: Flag and avoid changes to deprecated per-org installation mode content (ADR 0044)
internal/config/config.go[311-312]
internal/cli/admin.go[396-405]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Adding `codex` to the shared runtime list unintentionally enables it for deprecated per-org installations.

## Issue Context
The per-org branch of `fullsend admin install` validates against `config.ValidRuntimes()`. Keep Codex available for supported per-repo and per-agent paths while preventing this shared enumeration from extending deprecated org mode.

## Fix Focus Areas
- internal/config/config.go[311-312]
- internal/cli/admin.go[396-405]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 72 rules
Review mode: 🧠 Deep: This is a broad, behavior-changing runtime enablement across CLI/configuration, credential and environment forwarding, sandbox hooks, stream parsing, and gated end-to-end tests, with many independent logic paths where a redundant review can catch subtle defects.

Grey Divider

Tip of the day
💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread internal/config/config.go
Comment thread internal/config/config.go
Comment thread internal/cli/runtime_prompt.go Outdated
Comment thread internal/cli/run_overrides.go
Comment thread internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 6:58 PM UTC · Ended 7:02 PM UTC

Commit: fe65f48 · View workflow run →

@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/cli/github.go 50.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 7:04 PM UTC · Ended 7:10 PM UTC

Commit: 8d51f21 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 7:12 PM UTC · Ended 7:18 PM UTC

Commit: 103b8d5 · View workflow run →

@waynesun09
waynesun09 force-pushed the codex-runtime-enable-docs branch 2 times, most recently from 933a5f1 to e14e3fd Compare September 2, 2026 19:17
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:19 PM UTC · Completed 7:41 PM UTC

Commit: e14e3fd · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.88

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

Risk Assessment: elevated (3/5)

Details

Elevated risk preserved from prior assessment. Tier 1 unchanged: 50 files, 1044 lines, large blast radius, 7 protected paths (CI workflows changed), test ratio 0.14. Tier 2 shows high 30-day churn and elevated fix/revert density on core runtime files, offset by very fresh code age, single-author contention, and zero sentiment/revert flags. Tier 3 is favorable: well-scoped issue #6920, PR matches checklist item E exactly, opt-in runtime selection via capability gate acts as a feature flag with default disabled.

Previous run

Risk Assessment: elevated (3/5)

Details

Elevated risk preserved from prior assessment. Tier 1 unchanged: 50 files, 1044 lines, large blast radius, 7 protected paths (6 CI workflows), CI workflows changed, test ratio 0.14. Tier 2 shows extreme 30-day churn and high fix/revert density on core files like run.go and architecture.md, offset by very fresh code age and zero sentiment flags. Tier 3 is favorable: well-scoped issue #6920, PR matches checklist item E exactly, opt-in runtime selection acts as a feature gate. Weighted composite at the 2/3 boundary; the structural risk from protected-path count, blast radius, and high file churn on core CLI and config paths sustains the elevated level.

Previous run (2)

Risk Assessment: elevated (3/5)

Details

Elevated risk unchanged from prior assessment. Tier 1 signals identical (50 files, 1044 lines, 7 protected paths including 6 CI workflows, large blast radius). High churn on core files offset by very recent code age, additive new files, and opt-in runtime selection acting as a feature gate. Well-structured 5-PR stack with clear issue scope (#6920), non-first-time author, test coverage present (0.14 ratio).

Previous run (3)

Risk Assessment: elevated (3/5)

Details

Elevated risk unchanged from prior assessment. Tier 1 signals identical (50 files, 1044 lines, 7 protected paths including 6 CI workflows, large blast radius). High churn on core files (config.go, run.go, registry.go) offset by additive new files and documentation. Well-structured 5-PR stack with clear issue scope; this PR slice focuses on config/docs/CLI rather than security-sensitive credential plumbing. Non-first-time author, test coverage present (0.14 ratio).

Previous run (4)

Risk Assessment: elevated (3/5)

Details

Elevated risk driven by large blast radius (50 files, 7 protected paths including 6 CI workflows), large change size (1042 lines), and active churn on key files. Mitigated by non-first-time author, additive feature with clear rollback path, well-defined issue scope (Part E of 5-PR stack), and test coverage present (7 test files, 0.14 ratio).

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 2, 2026

Copy link
Copy Markdown

Review

Findings

Medium

  • [protected-path] .github/workflows/reusable-{code,fix,prioritize,retro,review,triage}.yml, images/README.md — 7 files under protected paths (.github/, images/) are modified. The PR links to issue Track Codex (openai/codex) as a supported agent runtime — same secretless GPT path as pi, pinned in the sandbox image #6920 and explains the rationale: the workflow changes fix a pre-existing bug where FULLSEND_REPO_VARS was not forwarded on the standalone reusable-workflow route, and images/README.md updates the codex description from "stub" to "opt-in". Human approval is required for protected-path changes regardless of context.

  • [stale-doc] docs/cli/inference.md:7 — The opening paragraph says OpenAI WIF enrols repositories "for GPT models on the pi runtime" — codex also uses the same OpenAI WIF credential path but is not mentioned. The PR updated the summary table in docs/cli/README.md to say "pi or codex" but left inference.md's own opening paragraph unchanged.
    Remediation: Change "for GPT models on the pi runtime" to "for GPT models on the pi or codex runtime".

Low

  • [stale-reference] internal/cli/run.go:1036 — The user-facing warning message says model %q has a provider/id form, which is pi's — but with codex now selectable, provider/id is also codex's model form (openai/<id>). The comment on line 1032 has the same issue. The logic itself is correct; only the wording is stale.
    Remediation: Change the comment and warning to mention both pi and codex.

  • [stale-doc] docs/guides/README.md:20 — The OpenAI Workload Identity guide summary says "Run GPT models on the pi runtime without a stored OpenAI key" — codex uses the same credential path and should be mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [incomplete-doc] docs/runtimes/claude.md:76 — The "See also" section cross-references only Pi but does not link to the new Codex runtime page at runtimes/codex.md.
    Remediation: Add - [Codex](codex.md) — OpenAI models via the same secretless credential path to the See also list.

  • [stale-doc] docs/guides/getting-started/getting-inference.md:16 — Says "GPT models on the pi runtime use OpenAI Workload Identity Federation" — codex uses the same path but is not mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [stale-doc] docs/runtimes/claude.md:24 — Fallback chains paragraph says "pi reports it as unsupported and ignores it" without mentioning that codex also ignores fallback chains with a warning (as documented in the new runtimes/codex.md).
    Remediation: Change "pi reports it as unsupported and ignores it" to "pi and codex report it as unsupported and ignore it".

  • [naming-consistency] pkg/behaviourtest/steps/runtime_test.go:75 — Test function is named TestGivenPiAgent_CommitsDefinitionWithFixtureInlined but the function under test was renamed from givenPiAgent to givenRuntimeAgent. The test body already calls givenRuntimeAgent.
    Remediation: Rename to TestGivenRuntimeAgent_CommitsDefinitionWithFixtureInlined.

  • [stale-reference] pkg/behaviourtest/steps/runtime.go:31 — Doc comment on registerRuntimeSteps says "(pi today — see features/runtime/pi.feature)" but the PR adds codex as a second real runtime with its own feature file (codex-openai.feature).
    Remediation: Update to mention codex or generalize.

  • [stale-reference] internal/cli/admin.go:630 — The --runtime flag help text reads (claude, pi or dummy; dummy is for behaviour test orgs only) and does not mention codex. Every other --runtime flag in the PR was updated. Note: admin.go is part of the deprecated per-org installation mode (ADR 0044), so this is an informational gap in deprecated code rather than an active user-facing issue.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run

Review

Findings

Medium

  • [protected-path] .github/workflows/reusable-{code,fix,prioritize,retro,review,triage}.yml, images/README.md — 7 files under protected paths (.github/, images/) are modified. The PR links to issue Track Codex (openai/codex) as a supported agent runtime — same secretless GPT path as pi, pinned in the sandbox image #6920 and explains the rationale: the workflow changes fix a pre-existing bug where FULLSEND_REPO_VARS was not forwarded on the standalone reusable-workflow route, and images/README.md updates the codex description from "stub" to "opt-in". Human approval is required for protected-path changes regardless of context.

  • [stale-doc] docs/cli/inference.md:7 — The opening paragraph says OpenAI WIF enrols repositories "for GPT models on the pi runtime" — codex also uses the same OpenAI WIF credential path but is not mentioned. The PR updated the summary table in docs/cli/README.md to say "pi or codex" but left inference.md's own opening paragraph unchanged.
    Remediation: Change "for GPT models on the pi runtime" to "for GPT models on the pi or codex runtime".

Low

  • [stale-reference] internal/cli/admin.go:630 — The --runtime flag help text reads (claude, pi or dummy; dummy is for behaviour test orgs only) and does not mention codex. Every other --runtime flag in the PR was updated. Note: admin.go is part of the deprecated per-org installation mode (ADR 0044), so this is an informational gap in deprecated code rather than an active user-facing issue.
    Remediation: Update the help text at line 630 to include codex.

  • [stale-reference] internal/cli/run.go:1036 — The user-facing warning message says model %q has a provider/id form, which is pi's — but with codex now selectable, provider/id is also codex's model form (openai/<id>). The comment on line 1032 has the same issue. The logic itself is correct; only the wording is stale.
    Remediation: Change the comment and warning to mention both pi and codex.

  • [stale-doc] docs/guides/README.md:20 — The OpenAI Workload Identity guide summary says "Run GPT models on the pi runtime without a stored OpenAI key" — codex uses the same credential path and should be mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [incomplete-doc] docs/runtimes/claude.md:76 — The "See also" section cross-references only Pi but does not link to the new Codex runtime page at runtimes/codex.md.
    Remediation: Add - [Codex](codex.md) — OpenAI models via the same secretless credential path to the See also list.

  • [stale-doc] docs/guides/getting-started/getting-inference.md:16 — Says "GPT models on the pi runtime use OpenAI Workload Identity Federation" — codex uses the same path but is not mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [stale-doc] docs/runtimes/claude.md:24 — Fallback chains paragraph says "pi reports it as unsupported and ignores it" without mentioning that codex also ignores fallback chains with a warning (as documented in the new runtimes/codex.md).
    Remediation: Change "pi reports it as unsupported and ignores it" to "pi and codex report it as unsupported and ignore it".

  • [naming-consistency] pkg/behaviourtest/steps/runtime_test.go:77 — Test function is named TestGivenPiAgent_CommitsDefinitionWithFixtureInlined but the function under test was renamed from givenPiAgent to givenRuntimeAgent. The test body already calls givenRuntimeAgent.
    Remediation: Rename to TestGivenRuntimeAgent_CommitsDefinitionWithFixtureInlined.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (2)

Review

Findings

Medium

  • [protected-path] .github/workflows/reusable-{code,fix,prioritize,retro,review,triage}.yml, images/README.md — 7 files under protected paths (.github/, images/) are modified. The PR links to issue Track Codex (openai/codex) as a supported agent runtime — same secretless GPT path as pi, pinned in the sandbox image #6920 and explains the rationale: the workflow changes fix a pre-existing bug where FULLSEND_REPO_VARS was not forwarded on the standalone reusable-workflow route, and images/README.md updates the codex description from "stub" to "opt-in". Human approval is required for protected-path changes regardless of context.

  • [stale-doc] docs/cli/inference.md:7 — The opening paragraph says OpenAI WIF enrols repositories "for GPT models on the pi runtime" — codex also uses the same OpenAI WIF credential path but is not mentioned. The PR updated the summary table in docs/cli/README.md to say "pi or codex" but left inference.md's own opening paragraph unchanged.
    Remediation: Change "for GPT models on the pi runtime" to "for GPT models on the pi or codex runtime".

Low

  • [stale-reference] internal/cli/admin.go:630 — The --runtime flag help text reads (claude, pi or dummy; dummy is for behaviour test orgs only) and does not mention codex. Every other --runtime flag in the PR was updated. Note: admin.go is part of the deprecated per-org installation mode (ADR 0044), so this is an informational gap in deprecated code rather than an active user-facing issue.
    Remediation: Update the help text at line 630 to include codex.

  • [stale-reference] internal/cli/run.go:1038 — The user-facing warning message says model %q has a provider/id form, which is pi's — but with codex now selectable, provider/id is also codex's model form (openai/<id>). The comment on line 1034 has the same issue. The logic itself is correct; only the wording is stale.
    Remediation: Change the comment and warning to mention both pi and codex.

  • [stale-doc] docs/guides/README.md:20 — The OpenAI Workload Identity guide summary says "Run GPT models on the pi runtime without a stored OpenAI key" — codex uses the same credential path and should be mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [incomplete-doc] docs/runtimes/claude.md:76 — The "See also" section cross-references only Pi but does not link to the new Codex runtime page at runtimes/codex.md.
    Remediation: Add - [Codex](codex.md) — OpenAI models via the same secretless credential path to the See also list.

  • [stale-doc] docs/guides/getting-started/getting-inference.md:16 — Says "GPT models on the pi runtime use OpenAI Workload Identity Federation" — codex uses the same path but is not mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [naming-consistency] pkg/behaviourtest/steps/runtime_test.go:77 — Test function is named TestGivenPiAgent_CommitsDefinitionWithFixtureInlined but the function under test was renamed from givenPiAgent to givenRuntimeAgent. The test body already calls givenRuntimeAgent.
    Remediation: Rename to TestGivenRuntimeAgent_CommitsDefinitionWithFixtureInlined.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (3)

Review

Findings

Medium

  • [protected-path] .github/workflows/reusable-{code,fix,prioritize,retro,review,triage}.yml, images/README.md — 7 files under protected paths (.github/, images/) are modified. The PR links to issue Track Codex (openai/codex) as a supported agent runtime — same secretless GPT path as pi, pinned in the sandbox image #6920 and explains the rationale: the workflow changes fix a pre-existing bug where FULLSEND_REPO_VARS was not forwarded on the standalone reusable-workflow route, and images/README.md updates the codex description from "stub" to "opt-in". Human approval is required for protected-path changes regardless of context.

  • [stale-doc] docs/cli/inference.md:7 — The opening paragraph says OpenAI WIF enrols repositories "for GPT models on the pi runtime" — codex also uses the same OpenAI WIF credential path but is not mentioned. The PR updates the parallel text in many other doc files (docs/cli/github.md, docs/guides/infrastructure/openai-workload-identity.md, etc.) to say "pi or codex", but this file was missed.
    Remediation: Change "for GPT models on the pi runtime" to "for GPT models on the pi or codex runtime".

Low

  • [stale-reference] internal/cli/admin.go:630 — The --runtime flag help text reads (claude, pi or dummy; dummy is for behaviour test orgs only) and does not mention codex. Every other --runtime flag in the PR was updated. Note: admin.go is part of the deprecated per-org installation mode (ADR 0044), so this is an informational gap in deprecated code rather than an active user-facing issue.

  • [stale-doc] docs/guides/README.md:20 — The OpenAI Workload Identity guide summary says "Run GPT models on the pi runtime without a stored OpenAI key" — codex uses the same credential path and should be mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".

  • [incomplete-doc] docs/runtimes/claude.md:76 — The "See also" section cross-references only Pi but does not link to the new Codex runtime page at runtimes/codex.md.
    Remediation: Add - [Codex](codex.md) — OpenAI models via the same secretless credential path to the See also list.

  • [stale-doc] docs/guides/getting-started/getting-inference.md:16 — Says "GPT models on the pi runtime use OpenAI Workload Identity Federation" — codex uses the same path but is not mentioned.
    Remediation: Change "on the pi runtime" to "on the pi or codex runtime".


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (4)

Review

Findings

Medium

  • [protected-path] .github/workflows/reusable-{code,fix,prioritize,retro,review,triage}.yml, images/README.md — 7 files under protected paths (.github/, images/) are modified. The PR links to issue Track Codex (openai/codex) as a supported agent runtime — same secretless GPT path as pi, pinned in the sandbox image #6920 and explains the rationale: the workflow changes fix a pre-existing bug where FULLSEND_REPO_VARS was not forwarded on the standalone reusable-workflow route, and images/README.md updates the codex description from "stub" to "opt-in". Human approval is required for protected-path changes regardless of context.

Low

  • [stale-reference] internal/cli/admin.go:630 — The --runtime flag help text reads (claude, pi or dummy; dummy is for behaviour test orgs only) and does not mention codex. Every other --runtime flag in the PR was updated. Note: admin.go is part of the deprecated per-org installation mode (ADR 0044), so this is an informational gap in deprecated code rather than an active user-facing issue.

  • [stale-reference] internal/cli/agent.go:170 — The --model flag help text says provider/id on pi without mentioning codex. The sibling --runtime flag on line 169 was updated to include codex in this PR, and the --model flag in run.go was updated to on pi and codex, but agent.go was missed.
    Remediation: Update to "model for this agent (alias, model id, or provider/id on pi and codex — codex takes OpenAI ids only); \"\" clears it".

  • [pattern-inconsistency] internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env.sh:43 — The comment above override_keys still only references FULLSEND_PI_MODEL as the runtime-scoped model alias; it does not mention the newly added FULLSEND_CODEX_MODEL on line 45.

  • [pattern-inconsistency] internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env-test.sh:54 — Test cases 6b/6c (new codex model forwarding tests) create a numbering collision with the existing # 6. Fallback chain keeps commas test that follows them.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 3, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 12:49 AM UTC · Ended 1:11 AM UTC

Commit: 2d31a58 · View workflow run →

Base automatically changed from codex-runtime-core to main September 3, 2026 01:10
@waynesun09
waynesun09 force-pushed the codex-runtime-enable-docs branch from 2d31a58 to a098af9 Compare September 3, 2026 01:10
Adds "codex" to config.ValidRuntimes(), so an org, a repo or a single
agents: entry can select it the way pi has been selectable since #6464.
The stub added with the image pin was registered in Resolve() but kept out
of ValidRuntimes() precisely so nothing could reach it before Bootstrap and
Run existed; with the runtime implemented, that gate comes off.

Everything that enumerates runtimes derives from ValidRuntimes(), so the
error text in config validation, the manifest validator, run overrides,
`fullsend github setup`/`repos` and the behaviour-test step all pick codex
up for free. The two places that restate the list by hand — the --runtime
flag help and the CLI reference tables — are updated here, as are the
manifest comments naming the installable runtimes.

codex also joins the interactive runtime prompt: userRuntimeChoices()
filters ValidRuntimes() down to the human-selectable set, which is codex
now that it is no longer a stub. opencode stays a stub and stays rejected
through every config path.

Refs #6920

Assisted-by: Claude (implementation)
Signed-off-by: Wayne Sun <gsun@redhat.com>
Adds docs/runtimes/codex.md and wires codex into the pages that enumerate
runtimes, so a reader choosing one sees three options rather than two.

The page follows the shape the other runtime pages use — what it is, one
command, models, at-a-glance, running it locally, behaviour differences,
what is not yet exercised, troubleshooting — and states only codex's own
side; cross-runtime comparison stays in runtimes.md, and the hardening
rationale stays out of user docs entirely.

runtimes.md gains a codex row in the status table, a third column in the
comparison and config-key tables, and a models paragraph. "Choosing between
claude and pi" becomes "Choosing a runtime" now that it covers three, with
the old heading kept as an explicit anchor so existing links still land.

The local-run guide's OpenAI section covered "GPT on pi only"; codex takes
the same key and the same harness requirements, so the heading and its body
now name both runtimes — again with the old anchor preserved, and pi.md's
inbound link moved to the new one. The WIF guide's opening sentence gets the
same treatment; nothing else there is runtime-neutral yet, so nothing else
changed.

Several passages describe behaviour whose exact wording depends on the
runtime implementation's smoke evidence — the plan block transcript, the
preflight error text, the platform matrix, the exact troubleshooting
messages. Those carry TODO(D) markers rather than invented text. The links
to ADR 0099 and the contributing page's codex section are held the same
way: the markdown link checker rejects targets that do not exist yet.

Verified: lychee (offline, --include-fragments) reports 0 errors over 932
unique links, and `vitepress build docs` renders runtimes/codex.html with
no dead links.

Refs #6920

Assisted-by: Claude (implementation)
Signed-off-by: Wayne Sun <gsun@redhat.com>
Adds features/runtime/codex-openai.feature, the codex counterpart of
pi-openai.feature: the repo's `runtime: codex` reaches backend selection,
the runner exchanges the job's OIDC token into a run-scoped OpenShell
provider, and codex reaches api.openai.com holding nothing but the
gateway's placeholder. A failed exchange or a rejected placeholder fails
the workflow, so a successful run is the assertion.

Gated on `runtime-codex-openai` and deliberately not in the Makefile
default — it needs an OpenAI organization mapped to the pool repositories.
Codex has no Vertex path, so unlike pi (whose runtime-pi scenario runs on
every behaviour job) this is its *only* behaviour scenario: until that
organization exists codex has no default behaviour coverage at all, and its
evidence is unit tests, recorded fixtures and local smoke runs. The feature
header, the Makefile comment and the behaviour-testing guide all say so
rather than leaving it to be inferred from an absent capability.

Two step definitions:

  * `a codex agent "<name>" defined as:` — the agent-committing helper was
    never pi-specific (it commits frontmatter + body and inlines a result
    fixture), so it is renamed givenRuntimeAgent and both step wordings
    land on it.
  * `the codex output stream records at least one tool call` — asserts a
    completed command_execution item in the tee'd `codex exec --json`
    stream (output.jsonl). That artifact's shape is fullsend's own, unlike
    codex's rollout session files, and the check refuses to accept one of
    those by requiring a top-level codex ThreadEvent type.

Refs #6920

Assisted-by: Claude (implementation)
Signed-off-by: Wayne Sun <gsun@redhat.com>
…ULLSEND_CODEX_MODEL

`FULLSEND_PI_MODEL` was a single hard-coded case in resolveRunOverrides.
Codex needs the same thing, so the gate becomes a runtimeModelEnv(runtime)
lookup and the pi case is one entry in it.

Why codex needs one: it serves OpenAI models only and needs one named, while
the fleet harnesses all ship `model: opus`. Without a runner-side knob a repo
moving to codex would have to edit every harness it runs. FULLSEND_CODEX_MODEL
is that one place, and it mirrors the pi knob exactly — read only when codex
is the runtime actually selected, below --model and FULLSEND_MODEL and above
the agents: entry and the harness. It is a per-repo default, not an override,
and when it decides the model the plan block and metrics.json's
override_source name it like any other source.

The lookup deliberately lists only the runtimes that need a knob rather than
deriving FULLSEND_<NAME>_MODEL for all of them: that would document variables
nobody asked for on claude and the test runtimes, and dummy-playback could
not have one at all — a hyphen is not legal in an environment variable name.

No new plumbing. run.go already resolves the config runtime and re-runs the
override resolution so a runtime-scoped alias can be gated on the selected
runtime; that path was written for pi and generalises unchanged. The
precedence tests now cover both knobs and that neither leaks into the other's
runtime.

Typed `refactor` rather than `feat`: the substance is the generalisation, and
the release-notes-worthy capability in this PR is the selectable codex runtime
itself, not its environment variable (COMMITS.md, "`feat` is for end users").

Also corrects a drift the knob exposed: the --model flag help and its row in
docs/cli/run.md both said `provider/id` applies "on pi", which is now true of
codex too.

Refs #6920

Assisted-by: Claude (implementation)
Signed-off-by: Wayne Sun <gsun@redhat.com>
Replaces every TODO(D) placeholder in docs/runtimes/codex.md with what the
runtime actually does, now that the codex runtime and ADR 0099 are on the
branch. Nothing here is paraphrased: the messages are quoted from
codex_run.go and codex_bootstrap.go, the behaviour from the smoke evidence
recorded in the contributing doc.

  * The model refusal is the real text, and the alias case is the one shown
    — it is the failure a repo on fleet harnesses actually hits. Both other
    cases (foreign prefix, no model at all) are named as failing the same
    way with the same two fixes.
  * The plan block gains the Agent/exit lines, with a note that the version
    on it is the codex CLI from Bootstrap's preflight, not the model.
  * A stale image is named by its real preflight failure,
    `codex --version` exited 127, and the version floor is stated the way
    pi.md did before its release shipped.
  * The hook-guard entry quotes the exit-97 message and mentions the sibling
    config-pinning guard, which is a credential-leak guard rather than a
    hook one.
  * Two entries added from the smoke run: the denied `GET /v1/models` at
    startup is expected and non-fatal (codex probes the catalog; the profile
    allows only POST /v1/responses), and a model the account cannot serve
    fails as reconnects plus a turn.failed carrying the 404 rather than at
    startup — which is why the verdict comes from the stream, not the exit
    code.

Two behaviour bullets were wrong and are corrected rather than expanded. A
PostToolUse block withholds the tool output from the model entirely; it does
not pass it through with a note, which is what the page claimed. And the
repository's own `.agents/skills` *are* discovered — leaving the project
untrusted suppresses `.codex/` config, not skills — so the page now says so
and adds that they are covered by the content scans and that codex's bundled
skills are switched off. The artifact-side caveat is stated too: output.jsonl
keeps raw command output either way, since the hooks protect the model's
context rather than the run's artifacts.

The links to ADR 0099 and the contributing page's codex section are un-held
now that both targets exist.

Refs #6920

Assisted-by: Claude (implementation)
Signed-off-by: Wayne Sun <gsun@redhat.com>
…docs

Grok and Codex sol review of the codex enablement PR.

The critical one is a real hole in the feature this PR adds. The documented
CI route for FULLSEND_CODEX_MODEL — set it as a repository variable — never
worked: setup-agent-env.sh's override_keys forwards FULLSEND_PI_MODEL and
FULLSEND_PI_PROVIDER but not the codex knob, so the value was silently
dropped before the agent job ever saw it. It is in the list now, with shell
and Go tests covering both the plain and the role-prefixed name.

Codex was accepted by config but invisible or misdescribed in most places a
person would look:

  * the interactive `github setup` prompt offered only [claude] and [pi]
    while accepting "codex". The menu lists it now, and the prompt test
    asserts every runtime the prompt accepts is also offered, so the next
    runtime cannot repeat this.
  * --runtime help on `agent`/`github setup`/`repos`, the setup PR body,
    docs/cli/{agent,repos,run}.md, and the getting-started guides all said
    "claude or pi".
  * `github setup --runtime codex` gave no stale-image warning; pi's had one.
  * registry.go, codex.go, runtime-implementation.md, sandbox-topology.md
    and images/README.md still called codex a stub kept out of
    ValidRuntimes() — which this PR is what changes.
  * architecture.md's runtime diagram had no codex node, though the config
    node above it listed the key.

Four documentation claims were wrong rather than merely missing, which is
the worse kind:

  * The effort table said max maps to xhigh. D's mapping is the identity —
    an earlier draft did remap it and the change was reverted.
  * `tools:` was described as enforced by the pre-tool hook. It is not: a
    Bash(...) list is recorded and not enforced on codex, entries with no
    codex tool are dropped with a warning, and the tool-allowlist hook is
    opt-in behind FULLSEND_TOOL_ALLOWLIST. Overstating an enforcement
    control is the worst direction to be wrong in.
  * PostToolUse was described as withholding output whenever a sanitizer
    fires. Only a canary block withholds; the redaction, unicode and
    suppression stages cannot rewrite built-in tool output on codex and
    instead tell the model what the result contained.
  * Sub-agents were described as a missing tool. Codex has spawn_agent;
    what is missing is fullsend's persona roster for it.

Also: the guard failure messages and the model-refusal text are now quoted
from the code with their exit codes (127, 97, 97, 98) rather than
paraphrased; the artifacts row records that only uncompressed rollouts are
extracted and that codex-debug.log is redacted like the rest; the plan block
shows the model spec as the runner prints it; "opt-in per org" became "per
repo, per agent, or a repos.yaml default" in both the page and the
ValidRuntimes comment, since org mode is deprecated (ADR 0044); and the
OpenAI WIF guide names codex wherever it genuinely follows the same
credential path.

The behaviour step matched `"type":"command_execution"` as a substring,
which an item.started or a declined command satisfies just as well. It now
parses each line and requires an item.completed whose status is completed,
and isCodexStreamFile is structural for the same reason the production
parser is.

Refs #6920

Assisted-by: Claude (implementation), Grok (review), Codex gpt-5.6-sol (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
Grok's re-review of the enablement PR: the structural work was confirmed,
what was left is copy that still reads as though pi were the only alternative
to Claude Code.

Runtime enumerations: `defaults.runtime` values in the repos reference, the
valid `runtime` values in the layered-config reference, and the `runtime`
example in the run reference. `--model` help on `agent set` and `run` (and
both reference tables) said `provider/id` applies "on pi" — it applies on
codex too, with the constraint that codex takes OpenAI ids only, which the
help now states rather than leaving to the page.

OpenAI wording: the `--openai-*` flags, the CLI index, the operations guide
and the cli-internals tree all described the WIF identifiers as "GPT on pi".
They are the same identifiers for codex — it uses the same run-scoped
provider — so they now say "on pi or codex". Same for the run reference's
"OpenAI credentials on pi" heading and the `inference.openai` description.

The run reference's artifacts table listed only claude-debug.log and
pi-debug.log; codex-debug.log is a run artifact too.

behaviour-testing.md described the generic gated-scenario pattern but led
with pi's step name as though it were the only one. It now names both agent
steps and says they commit the same file, which is what the shared helper
does.

Two code comments were left asserting the old state: registry.go still
listed codex among the stub runtimes excluded from ValidRuntimes(), which
this PR is what changes, and the ValidRuntimes comment described pi as
opt-in "per org/repo" when org mode is deprecated (ADR 0044). The codex
stale-image warning now names fullsend-code alongside fullsend-sandbox, as
pi's does — both images carry the pin.

Two test-facing fixes: codexStreamLines ignored scanner.Err(), so a line
past the buffer or an unreadable file would have been reported as "the agent
ran no tools" — an assertion failure blaming the agent for a read problem —
and it now returns the error for the step to surface. And
TestResolveFromPerRepoConfig gained a positive `runtime: codex` case, so the
per-repo path is asserted to resolve the codex backend rather than only
asserted not to reject it.

Refs #6920

Assisted-by: Claude (implementation), Grok (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
… advisory

Codex sol's re-review of the enablement PR.

The positive org-config case I added — `Defaults.Runtime = "codex"` passing
validation — extended coverage of a deprecated surface. Org mode is on its
way out (ADR 0044), so codex's selectability is asserted where it is
actually used: the per-repo key and an `agents:` entry, both of which keep
their assertions. pi's existing org case is left as it is; removing it is
not this PR's business.

The other item was a wording problem with teeth. The runtime prompt and
three doc sentences said `review` and `retro` "stay on Claude Code" on
codex, which reads as something fullsend enforces. Nothing pins those
roles — a repo-wide `runtime: codex` applies to them like any other agent,
and they will run, just in a single context without the persona roster. The
prompt now offers it as a recommendation, and the codex page says plainly
that it is a recommendation rather than a restriction and names the way to
make it one (`runtime: claude` on those agents' entries), which is the part
a reader actually needs.

Refs #6920

Assisted-by: Claude (implementation), Codex gpt-5.6-sol (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
Qodo found that the documented repository-variable route only works on one
of the two ways a role workflow runs. reusable-dispatch.yml passes
FULLSEND_REPO_VARS into every setup-agent-env.sh step it drives, but the
standalone reusable-<role>.yml workflows — the ones a repo calls directly —
do not, so setup-agent-env.sh sees no repository variables at all and every
override is dropped before the agent job starts.

For codex that is the difference between working and not: the fleet
harnesses ship `model: opus`, and without FULLSEND_CODEX_MODEL reaching the
job the run is refused for naming a Claude model. The gap is not new and not
codex-specific — FULLSEND_MODEL, FULLSEND_EFFORT, FULLSEND_RUNTIME and pi's
knobs were dropped on the same route — but codex is what made it visible,
and it is one env line per workflow, mirroring what dispatch already does.

The GitLab CI template sets these variables itself rather than calling
setup-agent-env.sh, so it is unaffected.

Refs #6920

Assisted-by: Claude (implementation), Qodo (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
…x page

Review bot findings plus a UX read of docs/runtimes/codex.md.

`fullsend agent set --model` help still said `provider/id` applies "on pi";
it matches `fullsend run`'s wording now, including that codex takes OpenAI
ids only. setup-agent-env.sh's comment above override_keys named only
FULLSEND_PI_MODEL when the list carries both knobs. The shell test's codex
cases were numbered 6b and 6c against an existing "# 6." further down; the
cases are sequential again, which meant renumbering the pre-existing 6a too
rather than leaving one oddity to explain the other.

The page itself was 296 lines against pi.md's 214, and the excess was
design notes rather than steps. Reading it as someone adopting codex for
the first time:

  * The PostToolUse bullet explained the hook pipeline in its own
    vocabulary — canary hits, redaction and unicode and suppression stages.
    A reader wants to know that a tool result is either withheld or
    annotated but never edited, and that artifacts are scrubbed regardless.
  * The skills bullet walked through where each scan happens. What changes
    for the reader is that skills work as they do on Claude Code and that
    codex's bundled ones are off.
  * "Not yet exercised" was three paragraphs of provenance. It now leads
    with the advice — start on a disposable repo, keep review and retro on
    Claude Code — and gives the pin as a config snippet instead of prose,
    with the version-by-version run record left in the contributing page,
    which already carries it.
  * The permission bullet argued the security model; it now states it.

Nothing accurate was dropped: the quoted allowlist warning, the
recommendation-not-restriction wording on review/retro, and the untested
WIF path all survive, in fewer words.

Refs #6920

Assisted-by: Claude (implementation), fullsend review bot (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
…ectable

PR #6923 added a TestAgentSettings_Validate row asserting an agents: entry
cannot select codex while it was a stub. This PR adds codex to
ValidRuntimes(); the positive coverage lives in
TestPerRepoConfigValidate_Runtime, TestResolveFromPerRepoConfig and
TestResolveForAgent_RejectsStubRuntimes, so the rejection row goes.

Refs #6920

Assisted-by: Claude (implementation)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@waynesun09
waynesun09 force-pushed the codex-runtime-enable-docs branch from a098af9 to 5bf423c Compare September 3, 2026 01:11
@waynesun09
waynesun09 dismissed fullsend-ai-review[bot]’s stale review September 3, 2026 01:12

Outdated: the only finding (test name) is a cosmetic LOW already handled on the branch; all threads resolved; dismissed so the last PR of the stack can enter the merge queue.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 3, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:13 AM UTC · Completed 1:30 AM UTC

Commit: 5bf423c · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $9.79

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The following review comments could not be posted on the diff (GitHub returned 422) and are included here instead:

  • internal/cli/run.go (file-level): Line 1036 · [low] stale-reference

The user-facing warning message says 'model %q has a provider/id form, which is pi's' — but with codex now selectable, provider/id is also codex's model form. The comment on line 1032 has the same issue. The logic itself is correct; only the wording is stale.

Suggested fix: Change the comment and warning to mention both pi and codex.

  • pkg/behaviourtest/steps/runtime_test.go (file-level): Line 75 · [low] naming-consistency

Test function is named TestGivenPiAgent_CommitsDefinitionWithFixtureInlined but the function under test was renamed from givenPiAgent to givenRuntimeAgent. The test body already calls givenRuntimeAgent.

Suggested fix: Rename to TestGivenRuntimeAgent_CommitsDefinitionWithFixtureInlined.

  • pkg/behaviourtest/steps/runtime.go (file-level): Line 31 · [low] stale-reference

Doc comment on registerRuntimeSteps says '(pi today — see features/runtime/pi.feature)' but the PR adds codex as a second real runtime with its own feature file (codex-openai.feature).

Suggested fix: Update to mention codex or generalize.

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Merged via the queue into main with commit b498a8c Sep 3, 2026
53 checks passed
@waynesun09
waynesun09 deleted the codex-runtime-enable-docs branch September 3, 2026 01:38
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 3, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 1:39 AM UTC · Completed 1:55 AM UTC

Commit: 5bf423c · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.99

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #6927feat(config): make the codex runtime selectable

Timeline

PR #6927 was a large, well-documented 50-file PR by waynesun09, the final piece of a 5-PR stack enabling the codex runtime (#6920). It was created on 2026-09-02, approved by ralphbean, and merged by the author on 2026-09-03 after ~7 hours.

The review agent (fullsend-ai-review) ran 8 times on this PR: 2 completed successfully (~22 and ~18 minutes), 1 was cancelled after 23 minutes mid-execution (superseded by a concurrent push), 3 were cancelled almost immediately (superseded), and 2 had the review job skipped. Both successful reviews issued a REQUEST_CHANGES verdict with only low-severity findings (stale references, naming consistency). The qodo-code-review bot independently found the PR's actual High-severity bug: standalone reusable workflows not forwarding FULLSEND_REPO_VARS, which the author promptly fixed.

Observations mapped to existing issues

Most improvement opportunities from this PR are already well-tracked:

What went well

  • The PR was exceptionally well-documented with clear commit messages, issue linkage, and verification evidence.
  • The qodo bot found the actual bug, and the author addressed it promptly.
  • The review agent correctly identified all commented files as files within the PR diff (no out-of-diff false positives).
  • The concurrency group mechanism correctly prevented multiple simultaneous reviews from completing.
  • The risk/elevated label was appropriately applied given the PR's blast radius.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk/elevated PR risk: elevated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants