Skip to content

Add opt-in strict JSON Schema output for OpenAI models, closes #1661 - #1677

Open
wellparth wants to merge 1 commit into
simonw:mainfrom
wellparth:main
Open

wellparth wants to merge 1 commit into
simonw:mainfrom
wellparth:main

Conversation

@wellparth

Copy link
Copy Markdown

Closes #1661.

Adds an opt-in strict_schema model option for OpenAI models that enables Structured Outputs with strict: true:

llm -m gpt-4o -o strict_schema 1 --schema 'name, age int' 'invent a cool dog'
model.prompt(prompt, schema=OutputModel, strict_schema=True)

What changed

  • New strict_schema option available on all OpenAI model options classes. Defaults to false so existing behaviour is unchanged. Setting it without a schema raises a clear error.
  • Chat Completions path (_Shared.build_kwargs): adds "strict": true to the json_schema response_format.
  • Responses API path (_SharedResponses._build_responses_kwargs): sets "strict": true instead of the previous hardcoded false.
  • New llm/default_plugins/openai_strict_schema.py: converts a JSON schema (e.g. one produced by a Pydantic model via model_json_schema()) into the restricted strict-compatible form:
    • adds additionalProperties: false to every object,
    • adds every property to required,
    • inlines $defs/$ref definitions,
    • preserves Optional-style anyOf: [T, {type: null}] unions,
    • raises a clear UnsupportedStrictSchemaError for keywords OpenAI strict mode cannot enforce (minItems, maxItems, allOf, multipleOf, etc.), rather than silently weakening the schema.

Notes on response handling

The issue also mentions surfacing finish_reason / refusal / truncation status. That's a larger change to llm.Response and is intentionally left out of this PR to keep it focused; the compiler/migration pieces here are the prerequisite for it.

Tests

  • tests/test_openai_strict_schema.py — converter unit tests (19): strict-shape normalization, $defs inlining, null unions, cosmetic-key stripping, error cases, input immutability.
  • tests/test_cli_openai_models.py — 6 integration tests using httpx2_mock: chat + responses bodies with/without the option, the requires-schema error, and the unsupported-keyword error.
  • Full suite: 1148 passed. ruff check and mypy clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add opt-in strict JSON Schema output for OpenAI models

1 participant