fix(ml,back): enforce the output token cap (#106) - #123
Merged
Conversation
…chat param (#106) `_GENERATION_PARAMS` declared `max_new_tokens: 512`, but the call path is `ModelInference.chat()`, whose schema (`TextChatParameters`) has no such field. Unknown keys are dropped in silence, so the cap was never in force and the service default of 1024 applied to both columns of the A/B. Measured against real Watsonx (eu-de, llama-3-3-70b) on the prompt "Write a detailed 800-word essay about the history of the printing press": max_new_tokens=16 -> TimeoutError on all 4 attempts (8s hard timeout) max_tokens=16 -> 15 words in 1.2s So the consequence was not only cost and latency: any prompt inviting a long answer exhausted the retries and surfaced as a 503 from `/api/generate`. Adds `backend/tests/test_generation_params.py` with a positive control — every key we send is checked against the dataclass fields of `TextChatParameters`, which is the check that would have caught this when it was introduced — plus a credential-gated live test that asks for far more than the cap allows and asserts the answer comes back cut short. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…udget
Measured against real Watsonx (eu-de, llama-3-3-70b, n=5 per cell, hard
timeout lifted so the true duration shows):
cap=512 vanilla median 4.5s worst 5.9s (~196 words)
autoria median 7.2s worst 8.4s (~313 words) <- over the cap
cap=320 vanilla median 5.0s worst 5.3s (~242 words)
autoria median 5.4s worst 5.6s (~223 words)
At 512 the conditioned branch straddles HARD_TIMEOUT_SECONDS: it sometimes
returns and sometimes exhausts all four attempts. A failed AutorIA branch
cannot be degraded away because the passport needs it, so that surfaces as
an intermittent 503 from POST /api/generate — reproduced twice while
verifying the RAG wiring end to end, once passing and once failing on the
same prompt.
320 keeps a margin under both the 8s hard timeout and the 10s client-side
abort in frontend/src/lib/api.ts, and still yields ~220 words per column.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 28, 2026
`test_generation_params.py` imports `autoria_ai.generator` to check our
parameters against the SDK schema. That passes locally, where an editable
install of `autoria_ai` sits in the venv, and fails in CI, which installs
only `backend/`:
ModuleNotFoundError: No module named 'autoria_ai'
Same resolution the production code already uses in
`app.routes.generate._ensure_ai_pipeline_on_path` — the repo-root
`ai_pipeline` directory goes on `sys.path`. This is exactly the local/CI
divergence the completeness audit warned about: a green local run proving
nothing about the deployed shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #106.
El defecto
_GENERATION_PARAMSdeclarabamax_new_tokens: 512, pero el camino de llamada esModelInference.chat(), cuyo esquema (TextChatParameters) no tiene ese campo. Las claves desconocidas se descartan en silencio, así que el tope nunca estuvo en vigor y se aplicaba el default del servicio: 1024.Medición contra Watsonx real (eu-de, llama-3-3-70b)
Prompt: "Write a detailed 800-word essay about the history of the printing press."
max_new_tokens=16(estado anterior)max_tokens=16(corregido)La consecuencia no era solo coste y latencia: cualquier prompt que invitara a una respuesta larga agotaba los reintentos y salía como 503 desde
/api/generate. Es más grave de lo que la issue estimaba.Prueba
backend/tests/test_generation_params.py:TextChatParameters. Es la comprobación que habría cazado esto el día que se introdujo, y caza cualquier futura clave mal escrita.Verificación
El único fallo del backend es
test_generate_live_watsonx, que apunta alibm/granite-3-8b-instructretirado upstream — ajeno a este cambio y en curso en el PR #120.🤖 Generated with Claude Code