Score level limits: the API enforces at most 10 levels, but the OpenAPI schema has no maximum, and a single level is accepted

[vbcherepanov]

Score level limits: the API enforces at most 10 levels, but the OpenAPI schema has no maximum, and a single level is accepted

What the docs say (API reference → ScoreQuestion): "A Score should have at least two levels; the API accepts up to 10."

What the OpenAPI schema says (https://api.typesafe.ai/openapi.json, ScoreQuestion.criteria): "type": "array", "minItems": 1, with no maxItems.

What the API does (checked 2026-09-21, jev-latest → jev-1.13.0):

for n in 1 2 10 11; do
  L=$(python3 -c "import json;print(json.dumps([f'level {i}' for i in range($n)]))")
  curl -s -w " HTTP %{http_code}\n" https://api.typesafe.ai/v1/systemone \
    -H "Authorization: Bearer $TYPESAFE_API_KEY" -H "Content-Type: application/json" \
    -d "{\"model\":\"jev-latest\",\"state\":\"The service is down for all users.\",\"questions\":{\"s\":{\"type\":\"score\",\"instructions\":\"How severe?\",\"criteria\":$L}}}"
done

levels	status	body
1	200	"score":0.0,"confidence":1.0,"probabilities":{"0":1.0}
2	200	normal answer
10	200	normal answer
11	400	{"detail":"Too many score levels. Must have at most 10 levels."}

Why it matters

• The maximum is enforced but absent from the schema, so clients generated from openapi.json cannot catch it, and it surfaces as a 400 that the error table does not list (see Docs report: request-body validation returns 400, but the API reference documents 422 #1).
• A one-level Score always returns confidence: 1.0 without looking at the state. It is accepted silently, so a caller who builds levels dynamically gets a confident, meaningless answer instead of an error.
• The SDKs disagree on the minimum: the Python SDK accepts one level (questions.py), the JS SDK requires two (questions.ts).

Suggested fix

• Add "maxItems": 10 to ScoreQuestion.criteria.
• Either reject fewer than two levels (and set "minItems": 2), or document that a single level is allowed and always returns confidence 1.0.

[aoright]

I cross-referenced this across the live OpenAPI schema (https://api.typesafe.ai/openapi.json), the official documentation, and the codebases of both official SDKs (typesafe-sdk-python and typesafe-sdk-js). Here is a consolidated audit of the current divergence:

1. Live OpenAPI Schema vs. API Runtime Enforcement

Fetching ScoreQuestion from https://api.typesafe.ai/openapi.json confirms:

"criteria": {
  "items": {
    "anyOf": [
      { "type": "string" },
      { "additionalProperties": true, "type": "object" },
      { "items": {}, "type": "array" }
    ]
  },
  "type": "array",
  "minItems": 1,
  "title": "Criteria"
}

• Upper bound: The schema specifies no maxItems. When 11 levels are sent, the FastAPI backend validator triggers HTTP 400 with {"detail": "Too many score levels. Must have at most 10 levels."}.
• Lower bound: The schema specifies minItems: 1. Sending a 1-level score returns HTTP 200 with trivial outputs (score: 0.0, confidence: 1.0, probabilities: {"0": 1.0}) without evaluating the input state.
• Accepted item types: Notice that criteria.items allows only string, object, and array (unlike ChoiceQuestion.criteria.additionalProperties which explicitly includes null).

2. Discrepancy Across SDK Implementations

• Python SDK (typesafe-sdk-python):
  • src/typesafe_sdk/_schemas/models.py:146 specifies min_length=1 with no upper bound:

    criteria: list[str | dict[str, Any] | list[Any]] = Field(..., min_length=1)

  • src/typesafe_sdk/_core/questions.py:77 (_validate_score_criteria) only checks if not criteria:.
  • As a result, 1-level scores pass locally, while 11-level scores pass locally and fail remotely with HTTP 400.
• JavaScript/TypeScript SDK (typesafe-sdk-js):
  • src/questions.ts:82 enforces:

    if (question.criteria.length < 2) {
      throw new TypeSafeError(
        `Score question "${name}" has ${question.criteria.length} criteria; at least two scores are required.`
      );
    }

  • It does not check question.criteria.length > 10.
  • As a result, 1-level scores are rejected locally with TypeSafeError, while 11-level scores pass locally and fail remotely with HTTP 400.

3. Recommendations

1. OpenAPI Schema:
   • Add "maxItems": 10 to ScoreQuestion.criteria.
   • Update minItems to 2 if scores semantically require at least two levels to establish an ordered scale, matching https://docs.typesafe.ai/api ("A Score should have at least two levels; the API accepts up to 10").
2. SDK Pre-flight Validation:
   • In typesafe-sdk-python: Add max_length=10 to ScoreQuestion.criteria in _schemas/models.py and check len(criteria) > 10 in _validate_score_criteria.
   • In typesafe-sdk-js: Add if (question.criteria.length > 10) check in validateQuestions in src/questions.ts to throw TypeSafeError before making the network request.