Skip to content

fix: reject an unrecognized language instead of falling back to English - #761

Merged
Colin Francis (colifran) merged 4 commits into
langchain-ai:mainfrom
easyhak:fix/reject-unrecognized-language
Aug 28, 2026
Merged

fix: reject an unrecognized language instead of falling back to English#761
Colin Francis (colifran) merged 4 commits into
langchain-ai:mainfrom
easyhak:fix/reject-unrecognized-language

Conversation

@easyhak

@easyhak easyhak commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

What & why

Passing a language name (e.g. Korean) instead of a BCP-47 code (ko) to openwiki_begin / --language used to print a warning and silently fall back to English.

The first call then persisted the run at en, and because resume refuses to change a started run's language (and force is not honored on that path), the only escape was manually deleting openwiki/.run.json and .last-update.json.

This is not hypothetical. It happened in two real project sessions.

session prompt first call argument
test1 /openwiki 초기화를 한국어로 진행해줘 language: "Korean"
test2 /openwiki … in korean language: "korean"

Both then hit the same dead end

The failure sequence unfolded like this


1. begin("Korean")                  → Silent fallback to "en", committing the run state to en.
2. Model notices: "Wait, it's in en?" → Retries with begin("ko").
3.                                  → Conflict error (interrupted run uses en ≠ ko).
4. Model tries force:               → begin("ko", force: true).
5.                                  → Conflict error again (force is ignored on this path).
6. The only escape:                 → Manually deleting .run.json and .last-update.json.

The fix

An unrecognized language is rejected before any run state or repository mutation exists, at every entry point. No wrong-language run can be persisted — on any host, or via raw --language — and a rejected request costs nothing but a retry.

The tool description carries its share of the work too, but the two do different jobs:

  • Tool description — prevention. Stops the bad value from being produced in the first place on Claude-family hosts.
  • Rejection — correctness. The actual guarantee. Holds regardless of which client calls, or whether it reads the description at all.

I measured both, because it was not obvious up front that the description alone would be enough (it isn't).

Experiment in Claude Code

40 isolated runs — fresh session and fresh temp git repo each — on Opus 5, Sonnet 5, and Haiku 4.5, using the real skill and --host claude, recording the actual JSON arguments the server received rather than the model's self-report.

Rigor notes
  • Both variants are clean full builds from source; compiled JS was never hand-edited.
  • The baseline's published tool surface (description + schema) was verified byte-for-byte against the globally installed 0.4.3 that caused the real incidents.
  • diff -rq over the whole dist confirmed the only difference between variants was the intended lines.
  • 5 runs per cell. Small, but the effect on Opus 5 is total (5/5 → 0/5) rather than marginal.

1. Does the description block the bad value?

3 models × 2 builds × 5 runs = 30. The "+ description" build adds only the description and schema hint — not the rejection. Metric: the language value the begin call passed. Zero timeouts.

model build passed values invalid
Opus 5 baseline "Korean" ×5 5/5
Opus 5 + description "ko" ×5 0/5
Sonnet 5 baseline "ko" ×4, "Korean" ×1 1/5
Sonnet 5 + description "ko" ×5 0/5
Haiku 4.5 baseline "ko" ×4, "한국어" ×1 1/5
Haiku 4.5 + description "ko" ×5 0/5

Baseline 7/15 invalid (47%) → 0/15 (0%). Fisher exact two-sided p = 0.0063; Opus alone p = 0.0079.

The strongest model(Opus) failed most: with no hint in the schema, "put the human-readable language name" is the natural reading.

2. Is a warning enough on its own?

Before settling on rejection I tried a response-side languageWarning, with the description reverted, and ran real begin calls on Opus 5 - checking the final .run.json language rather than just the argument.

build first call sequence final wiki language
warning only "Korean" ×5 Korean → ko → …, retried at once ko 2/5 · en 3/5
shipped (description + rejection) "ko" ×5 ko, done in one call ko 5/5

The warning brings the diagnosis forward but does not close the trap. Opus corrected to ko on the very next call every time, yet 3/5 still ended in English because the first call had already committed the run to en.

Noticing is not undoing.
That is why this PR rejects rather than warns.

Changes

  • src/platform/language.ts — replace { language?, warning? } with a three-case ResolvedLanguage union (absent | resolved | unrecognized), so a typo can no longer be collapsed into "no language requested". The type change alone surfaced every call site that was silently dropping the warning. Adds requireResolvedLanguage() as a post-boundary guard that throws if an unrecognized value ever slips past an entry point.
  • src/generation/repository-run.ts — reject an unrecognized language with invalid_input at the top of beginRepositoryRun, ahead of both ensureCodeModeRepoSetup and the resume branch, so neither a fresh nor a resumed run can be started with one.
  • src/cli/commands.tsparseRunCommand exits 1 on an unrecognized locale instead of surfacing a warning; languageWarning is removed from CliCommand. The -l, --language help text now names the expected BCP-47 form.
  • src/cli/cli.tsx — drop the now-unused stderr warning write.
  • src/integrations/core/protocol.ts, session-manager.ts — document that language is a BCP-47 code and that an unrecognized value fails with invalid_input and starts no run.

Behavior that is deliberately unchanged: a recognized language that differs from an interrupted run's still returns conflict. This PR only removes the case where the run was committed to a language the caller never asked for.

How tested

pnpm test (typecheck + build + vitest coverage)

  • test/platform/language.test.ts — the three result cases, name-vs-code rejection (Korean, 한국어, english, xx), ISO 639-2 narrowing (korko), and requireResolvedLanguage().
  • test/generation/repository-run.test.ts — a rejected request writes no .run.json, an interrupted run stays resumable after a rejected resume, and a retry with a real code succeeds.
  • CLI / runner / run-context tests updated for the removed languageWarning.

Breaking change

Callers relying on the silent English fallback for an invalid locale now get an error instead:

  • openwiki_begininvalid_input, no run started
  • --language → exit 1

Fix: pass a real BCP-47 code (ko, zh-CN, pt-BR) and rerun.

**Changeset is marked patch.
Happy to switch it to minor if you'd rather signal it in the version. just say the word.

An unrecognized `language` resolved to English and was recorded in run
state, which resume refuses to change, so the only way out was deleting
OpenWiki's own state files. Both entry points now reject it before
anything is written, and ResolvedLanguage is a discriminated union so
callers can no longer collapse "unrecognized" into "absent".

The openwiki_begin tool description and schema now state that the value
must be a BCP-47 code, so callers pass `ko` rather than `Korean`, and the
`--language` help text names the same form.

`--language` with an unrecognized value exits 1 instead of generating an
English wiki.
@changeset-bot

changeset-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c02e439

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
openwiki Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@easyhak easyhak changed the title fix: unrecognized language fix: reject an unrecognized language instead of falling back to English Aug 28, 2026

@colifran Colin Francis (colifran) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, easyhak!

@colifran
Colin Francis (colifran) merged commit 97c6ef0 into langchain-ai:main Aug 28, 2026
9 checks passed
@github-actions github-actions Bot mentioned this pull request Aug 28, 2026
Brad Huffman (ppsplus-bradh) added a commit to ppsplus-bradh/openwiki that referenced this pull request Aug 28, 2026
Pick up 4 upstream fixes (unrecognized-language rejection langchain-ai#761, GitHub Copilot
non-GPT-5 streaming langchain-ai#744, Bedrock maxTokens langchain-ai#743, docs langchain-ai#759).

One conflict, in beginRepositoryRun (src/generation/repository-run.ts): upstream
langchain-ai#761 now resolves and rejects an unrecognized language before touching the
repository, right where the feature computes its skipRepoSetup guard and
noopScope. Kept both — the language rejection runs first, then the recursive
repo-setup guard and the subproject no-op scope.

Full suite: 3680 passed / 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants