Structured AI decisions with multiple providers and local response validation.
Use Choice, Noul, and Score to classify application state without parsing chat
prose. Evaluate several questions in one request with Gemini, Groq, OpenAI, or a
local Ollama server. Decisions remain probabilistic: valid JSON does not guarantee
that a classification is correct.
From a checkout:
pip install -e .For the published package (check its version before relying on changes on main):
pip install system-one-nativeAgent skill:
npx skills add RodrigoAlbe/system-one --globalClaude Code plugin:
claude plugin marketplace add RodrigoAlbe/system-one
claude plugin install system-oneSet GEMINI_API_KEY, or select a different provider explicitly.
from system_one import SystemOneClient, Choice, Noul, Score
client = SystemOneClient(provider="gemini")
response = client.evaluate(
state={"message": "Payment gateway returning 500 errors", "plan": "Enterprise"},
questions={
"department": Choice(options=["Backend", "Billing", "DevOps"], instructions="Routing department"),
"urgent": Noul(instructions="Is there an active revenue-impacting outage?"),
"severity": Score(levels=["P1", "P2", "P3", "P4"], instructions="Operational severity"),
},
)
print(response.answers["department"].value)
print(response.answers["urgent"].value) # Model-estimated number in [0, 1]
print(response.metrics.output_tokens)
print(response.metrics.estimated_cost_usd) # None: unknown, not zeroKeyword arguments avoid confusion: positional constructors are
Choice(options, instructions) and Score(levels, instructions).
Convenience methods and async evaluation use the same validation:
category = client.choice("I love it", "Sentiment", ["Positive", "Neutral", "Negative"])
probability = client.noul("Unsolicited sales email", "Is this spam?")
priority = client.score("Disk almost full", "Severity", ["Low", "Medium", "High"])
# Inside an async function:
# response = await client.evaluate_async("log payload", {"alert": Noul("Page on-call?")})Every answer must contain exactly the expected fields. Missing/extra question
IDs, duplicate JSON keys, out-of-range values, booleans masquerading as numbers,
unknown options, non-finite numbers, and malformed JSON raise
InvalidResponseError. A batch is atomic: no partial result is returned.
Question options/levels must be non-empty lists of unique, non-empty strings.
from system_one import InvalidResponseError, ProviderError
try:
result = client.evaluate("message", {"urgent": Noul("Is this urgent?")})
except InvalidResponseError:
print("No usable decision: send to manual review")
except ProviderError:
print("Provider unavailable: queue for a later attempt")
else:
print(result.answers["urgent"].value)ProviderRefusalError and IncompleteResponseError are subclasses of
InvalidResponseError. Refused or truncated responses never become default
negative answers. Invalid responses are not automatically retried.
HTTP 408/429/500/502/503/504 and transport failures are retried with backoff,
respecting Retry-After. max_retries=3 retains its historical meaning of three
total attempts. timeout is an HTTP operation timeout, not a total deadline;
retries and provider-requested waits can increase overall latency.
| Provider | Environment variable | Default model | Automatic response mode |
|---|---|---|---|
| Gemini | GEMINI_API_KEY |
gemini-3.1-flash-lite |
JSON Schema via responseJsonSchema |
| Groq | GROQ_API_KEY |
llama-3.3-70b-versatile |
JSON object + schema in prompt |
| OpenAI | OPENAI_API_KEY |
gpt-4o-mini |
Strict JSON Schema |
| Ollama | None | qwen2.5:7b |
JSON Schema on the local OpenAI-compatible endpoint |
Auto-detection checks Gemini, Groq, then OpenAI keys. With no key it selects Gemini
and reports the missing key before making a request. For local inference use
SystemOneClient(provider="ollama") explicitly.
Groq openai/gpt-oss-20b and openai/gpt-oss-120b use strict schema mode. The known
OpenAI schema models are gpt-4o-mini, gpt-4o-mini-2024-07-18, and
gpt-4o-2024-08-06. Other OpenAI/Groq model names conservatively use JSON object
mode. Both modes include the complete schema in the prompt and validate locally.
For a custom model with verified support, set response_mode="json_schema".
Use response_mode="json_object" for older compatible servers. Overrides do not
make an unsupported model support a feature; provider errors remain explicit.
Provider contracts: Gemini API, Groq structured outputs, OpenAI structured outputs, Ollama structured outputs. Availability, quotas, latency, and billing depend on the provider, model, and account. This project does not guarantee free usage or sub-second latency.
confidence and Noul.value are model-reported estimates, not calibrated
probabilities of correctness. confidence_source is "model_reported".
raw_distribution remains None; truncated top-token alternatives cannot establish
an option-level distribution, especially across multiple questions or tokens.
use_logprobs=False is the default. Opt-in requests are supported for OpenAI and
Ollama endpoints that implement them; data is kept only in raw_response.
Groq and Gemini diagnostic opt-in currently raises ValueError locally.
Token logprobs never overwrite the JSON answer or its reported confidence.
The diagnostic helpers extract_openai_logprobs and extract_gemini_logprobs
require position= for multi-token responses. They preserve exact token text and
never pool probabilities across positions. entropy_confidence measures
concentration, not accuracy. Calibrate decision thresholds on independent labeled
data before using them for automated actions.
pip install -e ".[dev]"
python -m pytest
# Live calls; requires provider credentials and may incur usage charges:
python tests/benchmark.py --provider gemini --repeat 5 --output benchmark.json
python tests/benchmark.py --provider ollama --output local-benchmark.json
python tests/benchmark.py --provider groq --dataset cases.json --output labeled-benchmark.jsonThe three bundled scenarios are unlabeled smoke examples, not an accuracy study. The report includes model/configuration, timestamp, package/Python versions, dataset hash, per-request results, validation/provider failures, token usage, and successful-request latency p50/p95 (nearest-rank p95). Latency includes retries. Token totals cover successful responses only; failed requests may consume tokens. Cost remains unknown. No competitor or savings figures are fabricated.
A labeled dataset is a JSON array. Each case needs a unique id, state,
questions, and an expected label for every question:
[
{
"id": "example-only",
"state": "The server is unavailable to all users.",
"questions": {
"outage": {"type": "noul", "instructions": "Is the server unavailable?"}
},
"expected": {"outage": true}
}
]Use options for choice questions and levels for score questions; their labels
must be one of those strings. Noul labels are booleans. Reports add accuracy on
valid answers, correct answers over all attempted labels (including failures), and
Noul Brier score. Noul accuracy uses a 0.5 threshold. Repetitions do not increase
the number of independent cases. A representative, independently reviewed dataset
is still needed before claiming real-world quality or calibrated confidence.
- Invalid or missing answers now raise errors instead of becoming default values.
- Logprobs are disabled by default and no longer rewrite values or confidence.
- Unsupported diagnostic logprob requests fail before network access.
estimated_cost_usdis now optional and returnsNonewhen unknown.- Empty/duplicate options and invalid question definitions are rejected locally.
- Multi-position logprob extraction requires an explicit token position.
system-one noul "Customer requests a refund" "Is the customer requesting a refund?"
system-one choice "Server CPU at 99%" "Action" Scale Restart Ignore
system-one score "Database disk at 92%" "Severity" Low Medium High CriticalMIT.