Skip to content

revert: per-agent runtime/model routing (#1209) — fleet project lacks claude-sonnet-5 - #1223

Closed
waynesun09 wants to merge 1 commit into
mainfrom
revert-diverse-routing
Closed

revert: per-agent runtime/model routing (#1209) — fleet project lacks claude-sonnet-5#1223
waynesun09 wants to merge 1 commit into
mainfrom
revert-diverse-routing

Conversation

@waynesun09

Copy link
Copy Markdown
Member

What

Reverts the per-agent runtime/model routing merged today (#1209), restoring the previous config: every agent on Claude Code / opus, no models.aliases.

Why — the fleet is failing at the first model call

First fleet review after the merge (fullsend-ai/fullsend#7167, run 34365183694): Bootstrap resolved all nine personas, then the orchestrator's first request failed:

404 Publisher model `projects/…/locations/global/publishers/anthropic/models/claude-sonnet-5` was not found or your project does not have access
→ Result: ERROR · Validation failed: output/agent-result.json not found · Agent exit code: 1

The Vertex project the fleet runs on does not serve claude-sonnet-5. The alias sonnet: claude-sonnet-5 is on every sonnet parent (review, fix, triage, retro), and a pinned alias has no fallback, so every one of those agents fails on every trigger until this is reverted. code (Grok) and prioritize (Gemini) may or may not be served there — nothing has proven it on the fleet project, so the whole routing goes back until it is.

The local validation runs were done against a project that serves all three ids; the fleet's project is a different one. That is the mistake — the served-model check must be run against the project the fleet actually uses, which is the secret FULLSEND_GCP_PROJECT_ID, not a probe from a laptop.

Re-apply plan

  1. Confirm which ids the fleet project serves — a one-line fullsend run from CI against it, or an admin rawPredict probe from the project itself.
  2. Re-apply only the routing that project serves; keep sonnet on the default alias unless claude-sonnet-5 is enabled there.

… claude-sonnet-5

The first fleet review after #1209 merged failed at the orchestrator's
first model call: Vertex returned 404 for claude-sonnet-5 in the project
the fleet runs on (fullsend-ai/fullsend#7167, run 34365183694). The
sonnet alias sits on every sonnet parent and has no fallback, so review,
fix, triage and retro fail on every trigger. Restore the previous config
until the fleet project's served models are confirmed from CI.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@waynesun09
waynesun09 requested a review from a team as a code owner September 9, 2026 14:53
@waynesun09 waynesun09 added the fullsend-no-fix Skip bot-triggered fix agent runs label Sep 9, 2026
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Revert unsupported per-agent model routing

🐞 Bug fix ⚙️ Configuration changes 🕐 10-20 Minutes

Grey Divider

AI Description

• Restores a single Claude runtime and default opus model for every agent.
• Removes unsupported model aliases and per-agent routes that break fleet execution.
• Defers diverse routing until the fleet project's served models are verified.
Diagram

graph TD
  CFG["Repository config"] --> BOOT["Fullsend bootstrap"] --> RUNTIME["Claude runtime"] --> FLEET["All agents"] --> MODEL["Default opus"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Targeted Sonnet rollback
  • ➕ Preserves Grok and Gemini routing where available
  • ➕ Retains intended cost and specialization benefits
  • ➖ Fleet access to Grok and Gemini remains unverified
  • ➖ Leaves multiple failure paths during an active outage
2. Capability-gated routing
  • ➕ Enables routes only after fleet-project validation
  • ➕ Supports gradual reintroduction without another broad rollback
  • ➖ Requires probing or conditional configuration support
  • ➖ Adds complexity inappropriate for immediate recovery

Recommendation: Merge the full rollback as the safest immediate recovery because none of the newly routed model IDs are confirmed on the fleet project. Validate availability from CI using the actual fleet credentials, then reintroduce only proven routes; capability gating is preferable longer-term if the platform supports it.

Files changed (1) +0 / -28

Bug fix (1) +0 / -28
config.yamlRestore uniform Claude and opus routing +0/-28

Restore uniform Claude and opus routing

• Removes the 'sonnet' alias and all per-agent runtime, model, and subagent overrides. Every agent again inherits the repository-wide Claude runtime and default opus model, avoiding unsupported Vertex model requests.

.fullsend/config.yaml

@waynesun09
waynesun09 marked this pull request as draft September 9, 2026 14:54
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 9, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure (validation failed after 1 iteration(s)) · Started 2:55 PM UTC · Completed 2:57 PM UTC

Commit: b4f143f · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (1)

Grey Divider


Remediation recommended

1. Five review checks still use Sonnet 🐞 Bug ≡ Correctness
Description
Removing the review runtime and subagent overrides makes the restored Claude runner use the model
declared in each native subagent manifest, where five standard checks still specify sonnet rather
than inheriting opus. Whenever review dispatches the standard roster, those checks resolve through
the fleet pin claude-sonnet-4-6, so the revert does not deliver its stated all-Opus routing.
Code

.fullsend/config.yaml[L32-35]

-    - name: review
-      runtime: pi
-      model: sonnet
-      subagents:
Relevance

●●● Strong

Removing overrides restores the Claude runner but leaves native manifests' explicit Sonnet
selections, contradicting all-Opus intent.

PR-#1209

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The restored repository default is the Claude runtime and the review harness selects Opus, but
review dispatches the listed subagent roster and five corresponding manifests explicitly select
Sonnet. Repository documentation confirms Claude Code resolves subagent frontmatter aliases
independently, while the mounted fleet environment pins sonnet to Claude Sonnet 4.6.

.fullsend/config.yaml[7-16]
harness/review.yaml[1-11]
skills/pr-review/SKILL.md[36-50]
skills/pr-review/sub-agents/intent-coherence.md[1-7]
skills/pr-review/sub-agents/style-conventions.md[1-7]
skills/pr-review/sub-agents/docs-currency.md[1-6]
skills/pr-review/sub-agents/cross-repo-contracts.md[1-6]
skills/pr-review/sub-agents/risk-assessment.md[1-7]
docs/review.md[247-251]
env/gcp-vertex.env[6-17]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The reverted review configuration restores the Claude/Opus parent but leaves five native review subagent manifests selecting `sonnet`. Ensure every review subagent resolves to `opus` without reintroducing the unavailable Sonnet 5 alias.

## Issue Context
Claude Code resolves each subagent's frontmatter model independently from the parent model. Add explicit local Opus overrides for the complete review roster, or update the relevant manifests and pinned remote source so none of these checks retains Sonnet routing.

## Fix Focus Areas
- .fullsend/config.yaml[15-16]
- skills/pr-review/sub-agents/intent-coherence.md[1-7]
- skills/pr-review/sub-agents/style-conventions.md[1-7]
- skills/pr-review/sub-agents/docs-currency.md[1-6]
- skills/pr-review/sub-agents/cross-repo-contracts.md[1-6]
- skills/pr-review/sub-agents/risk-assessment.md[1-7]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

2. The revert cannot be corroborated 📜 Skill insight ⛨ Security
Description
The branch name matches revert-*, but neither the lowercase revert: PR title nor the commit
message matches the required Revert "..." form. Because the routing block is removed under
revert-only authorization, reviewers have no second prescribed metadata signal to corroborate that
classification.
Code

.fullsend/config.yaml[L12-14]

-models:
-    aliases:
-        sonnet: claude-sonnet-5
Relevance

● Weak

The PR title explicitly uses revert:, and its commit message clearly identifies the reverted
change.

PR-#1209

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1538397 requires at least two prescribed revert signals plus diff verification. The
supplied metadata provides only the matching branch name, while the diff removes the model-alias
routing block and the current configuration shows that routing is absent.

.fullsend/config.yaml[7-16]
Skill: pr-review

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR claims to be a revert but has only one of the required corroborating metadata signals: its `revert-*` branch name.

## Issue Context
Update either the PR title or commit message to the prescribed `Revert "..."` form, identify the original merged commit, and verify that these deletions are its inverse.

## Fix Focus Areas
- .fullsend/config.yaml[12-14]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 57 rules
✅ Skills: 4 invoked
  code-review
  code-implementation
  pr-review
  docs-review
✅ Cross-repo context — repo relationships
  Explored: repo: fullsend-ai/.fullsend (sha: 7c163cad)
  Explored: repo: fullsend-ai/fullsend (branch: agent/7000-reconcile-role-from-harness, sha: d0c3e3f8)
Review mode: ⚖️ Balanced: This is a runtime/model configuration revert affecting fleet-wide agent execution and external model availability, so it carries meaningful behavioral and operational risk despite the small diff.

Grey Divider

Tip of the day
💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread .fullsend/config.yaml
Comment on lines -32 to -35
- name: review
runtime: pi
model: sonnet
subagents:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Five review checks still use sonnet 🐞 Bug ≡ Correctness

Removing the review runtime and subagent overrides makes the restored Claude runner use the model
declared in each native subagent manifest, where five standard checks still specify sonnet rather
than inheriting opus. Whenever review dispatches the standard roster, those checks resolve through
the fleet pin claude-sonnet-4-6, so the revert does not deliver its stated all-Opus routing.
Agent Prompt
## Issue description
The reverted review configuration restores the Claude/Opus parent but leaves five native review subagent manifests selecting `sonnet`. Ensure every review subagent resolves to `opus` without reintroducing the unavailable Sonnet 5 alias.

## Issue Context
Claude Code resolves each subagent's frontmatter model independently from the parent model. Add explicit local Opus overrides for the complete review roster, or update the relevant manifests and pinned remote source so none of these checks retains Sonnet routing.

## Fix Focus Areas
- .fullsend/config.yaml[15-16]
- skills/pr-review/sub-agents/intent-coherence.md[1-7]
- skills/pr-review/sub-agents/style-conventions.md[1-7]
- skills/pr-review/sub-agents/docs-currency.md[1-6]
- skills/pr-review/sub-agents/cross-repo-contracts.md[1-6]
- skills/pr-review/sub-agents/risk-assessment.md[1-7]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 9, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:03 PM UTC · Completed 3:17 PM UTC

Commit: b4f143f · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $2.71

@fullsend-ai-review fullsend-ai-review Bot added the risk/low PR risk: low label Sep 9, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: low (1/5)

Details

Single-file, 28-line revert of a config change by a known non-bot human author with no protected/security-sensitive paths, no dependency or CI changes (Tier 1 composite ~1.1); Tier 2 churn/multi-author signals are moderate (~2.0) but no linked issue is present so weights redistribute to Tier1=62%/Tier2=38%, yielding a rounded composite score of 1 (low).

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Medium

  • [commit-convention-violation] N/A — The commit message and PR title use the type prefix revert:, which is not one of the Conventional Commit types this repo allows. COMMITS.md and AGENTS.md §4 restrict types to feat, fix, refactor, docs, test, chore, ci, perf, build; the same closed set is enforced by .gitlint ([contrib-title-conventional-commits] types=feat,fix,refactor,docs,test,chore,ci,perf,build) and by the commit-lint CI job in .github/workflows/lint.yml, which lints both the PR title and every commit subject. The commit-lint check on this PR's head commit is already reporting fail on GitHub, confirming this is not just a style nit but a merge-blocking CI failure. revert is not a recognized fallback here even though it's part of the wider Conventional Commits spec.
    Remediation: Rename the commit subject and PR title to use an allowed type, e.g. chore: revert per-agent runtime/model routing (#1209) — fleet project lacks claude-sonnet-5 or fix: revert per-agent runtime/model routing (#1209) — fleet project lacks claude-sonnet-5.

Everything else checks out: this is a verified, byte-exact revert of PR #1209's changes to .fullsend/config.yaml (the diff removes exactly the 28 lines that PR added — the models.aliases block and the 6 per-agent overrides — with no other changes and no leftover references elsewhere in the repo). No security, correctness, or documentation issues were found, and the change is self-authorizing as a clean revert (no linked issue needed).


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

@waynesun09

Copy link
Copy Markdown
Member Author

Not needed: the cause was the fleet project (prod) not serving claude-sonnet-5 / xai/grok-4.6, and those were enabled in its Model Garden on 2026-09-09. A natural fleet review then succeeded on the merged config (fullsend run 34367379698: nine personas resolved, seven persona-tagged dispatches across claude-sonnet-5, gemini-3.8-flash and xai/grok-4.6, cost $2.90). Closing without merge.

@waynesun09 waynesun09 closed this Sep 9, 2026
@waynesun09
waynesun09 deleted the revert-diverse-routing branch September 9, 2026 15:21
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 9, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 3:23 PM UTC · Completed 3:34 PM UTC

Commit: b4f143f · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.73

@fullsend-ai-retro

Copy link
Copy Markdown

Retro on #1223 — a human-authored revert of PR #1209's per-agent model routing, opened after the review dispatch on a downstream PR (fullsend-ai/fullsend#7167) hit a 404 because the fleet's production Vertex project didn't yet serve claude-sonnet-5. The revert itself never merged: the models were enabled in Vertex Model Garden the same day, a subsequent fleet review succeeded against the original #1209 config, and the author closed #1223 as moot. Two review agents commented — Qodo (external) and fullsend-ai-review (internal). I verified Qodo's claim that 5 review-subagent manifests 'still hardcode model: sonnet, defeating the revert' against actual repo state: the claim doesn't hold up — those subagents' effective model resolution is unaffected by the revert (3 were never touched by #1209 at all; the other 2 correctly fall back to their pre-#1209 manifest default once the config override is removed). The internal fullsend-ai-review agent's terse 'byte-exact revert, no correctness issues' conclusion was actually the accurate one here — a positive precision signal, not a gap. The internal review agent's other finding (commit message uses a revert: Conventional Commit prefix that .gitlint/AGENTS.md don't allow, failing the commit-lint CI gate) was correct and CI-confirmed, but this is a known, still-open recurrence: issue #924 (opened 2026-08-21 after PR #923 hit the identical failure) already tracks adding revert as an allowed commit type — I did not file a duplicate, but this is fresh evidence the gap is still live a PR later. I also found a cluster of prior open/closed issues (#1100, #1117 closed, #1186, #379, #1116 closed, #515, #256, #763) about the same underlying failure class — a model referenced in .fullsend/config.yaml routing not being served by the production GCP project — recurring across at least 4 separate incidents despite multiple prior fixes. None of the existing issues propose a deterministic pre-merge check against the real serving project, so I filed one new proposal for that. Autonomy-readiness assessment is inconclusive for this PR: it never merged and no human left review comments beyond the closing note, so there's no approval signal to evaluate for relaxing oversight.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fullsend-no-fix Skip bot-triggered fix agent runs risk/low PR risk: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant