fix: read the model tiers from the published policy instead of restating them - #76
Merged
Conversation
…ing them Every role, the advisor escalation, and the LLM policy prose fed to every factory agent each carried their own copy of a Claude model id. Three copies, no comparison between them, and the prose copy had gone wrong: it named `anthropic.claude-sonnet-4-6` as the default. That id cannot be invoked. Bedrock reports `inferenceTypesSupported: [INFERENCE_PROFILE]` for the whole current Claude family — there is no on-demand path — so a bare `anthropic.`-prefixed id is refused with a ValidationException on the first call. Every factory agent was being instructed to build against a form that does not work, and an app built to that instruction fails on its first Bedrock request. ─────────────────────────── One source ─────────────────────────── `llm-policy.json` joins `language-toolchain.json` as a vendored, digest-pinned copy of the published standard, and the tiers are read from it: - `LLM_MODELS` — the standard's `models`, as inference-profile ids. - `MODEL_TIERS` — the canonical short ids the roster declares, derived from `LLM_MODELS` by stripping the profile geo prefix, the `anthropic.` vendor prefix, and Bedrock's date/version suffix. - `LLM_POLICY` — assembled from the standard's own `models`, `regions_preferred`, `sdk_by_language` and `requirements` rather than restated. Rendering the requirements gained one the hand-written blob never had: `inference-profile-required`, which is the rule the blob was breaking. Nothing in fab's own prose had said the profile form was mandatory, which is why the drift went unremarked. `ADVISOR_MODEL` now reads `MODEL_TIERS.escalation`. It is the escalation tier by definition, and it was the copy most likely to be missed on a bump — the advisor is consulted mid-session rather than at deploy time, so a stale id there surfaces as a failed escalation inside a workflow rather than at startup. ──────────────────────── The current tier ──────────────────────── The roster moves to the Claude 5 family: 78 roles to `claude-sonnet-5`, 2 lab roles and the advisor to `claude-opus-5`. A role default is an invocation one level of indirection away, so a roster on a previous generation means every factory run infers on a lower tier than the policy names. `BEDROCK_MODEL_IDS` gains the two Claude 5 base ids. Without them a role on `claude-sonnet-5` throws "No AWS Bedrock model id is mapped" at session start under `FAB_INFERENCE=bedrock` — the regulated path. The suffixes in that table are irregular and no rule produces them: Opus 5 and Sonnet 5 are bare, Opus 4.6 carries `-v1`, Haiku 4.5 carries a release date and `-v1:0`. Every value was read off `aws bedrock list-foundation-models --by-provider anthropic`, and the comment now says so. Older entries stay. The table maps a canonical id to a Bedrock id; a caller may legitimately pin a role to an earlier model, and removing a mapping would break that rather than protect anything. ──────────────────────────── The gate ──────────────────────────── `__tests__/model-policy.test.ts` holds the three representations to each other: every role's model is a tier the policy names, no role sits on a bare foundation-model id, the advisor is the escalation tier, each tier resolves to a `us.anthropic.` profile id on bedrock and passes through unchanged on the api backend, and canonicalising a tier then resolving it returns the policy's exact id — which is the only check that catches a wrong suffix in either table. Mutation-tested seven ways: a role reverted to the previous generation, a role on a bare id, a policy bump with the Bedrock mapping forgotten, a wrong `-v1` suffix, the canonical-id derivation dropping the date suffix, the advisor pinned to a literal, and the prose reverting to a bare id. Each fails the case meant to catch it and no other. A model id is a plain string, so no compiler, linter, or type checker has an opinion about it. A test is the only thing that can. Also corrects the bare ids in the kagent-curation skill's ModelConfig example — invoke position, and the skill is what agents follow to author those configs. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three copies of the Claude model id lived in this repo — every role's
model, the advisor escalation, and the LLM policy prose fed to every factory agent — with nothing comparing them. The prose copy had drifted toanthropic.claude-sonnet-4-6.That id cannot be invoked. Bedrock reports
inferenceTypesSupported: [INFERENCE_PROFILE]for the whole current Claude family, so a bareanthropic.-prefixed id is refused with a ValidationException on the first call. Every factory agent was being instructed to build against a form that does not work.One source
llm-policy.jsonjoinslanguage-toolchain.jsonas a vendored, digest-pinned copy of the published standard.LLM_MODELS,MODEL_TIERS, and theLLM_POLICYprose are all read from it, andADVISOR_MODELreadsMODEL_TIERS.escalation.Rendering the standard's requirements gained one the hand-written blob never had:
inference-profile-required— the rule the blob was breaking. Nothing in fab's own prose had said the profile form was mandatory, which is why the drift went unremarked.The current tier
The roster moves to the Claude 5 family. A role default is an invocation one level of indirection away, so a roster on a previous generation means every factory run infers on a lower tier than the policy names.
BEDROCK_MODEL_IDSgains the two Claude 5 base ids — without them,FAB_INFERENCE=bedrockthrows at session start. The suffixes there are irregular and no rule produces them (Opus 5 bare, Opus 4.6-v1, Haiku 4.5 dated and-v1:0); every value was read offaws bedrock list-foundation-models.The gate
__tests__/model-policy.test.tsholds the three representations to each other, including that canonicalising a tier then resolving it returns the policy's exact id — the only check that catches a wrong suffix in either table.Mutation-tested seven ways; each fails the case meant to catch it and no other.
Verification
npm test484 pass ·npm run test:coverageexit 0 ·npm run lintexit 0 ·npm run format:checkclean ·npm run standards:checkboth files match the pin ·npm run buildbundles both standards intodist/.