Independent evaluation of Jev on Chinese classification tasks — 97.7% @ ~890 ms, fully deterministic across 3 runs (+ a runtime engine)

[lexingtonhibiki]

Hi — while building judgekit (MIT, a runtime judgment layer over the System One API: YAML-defined tasks → native choice/score/noul calls, OpenAI-compatible fallback, per-decision cost accounting), we ran a 4-provider comparison on 130 hand-labeled Chinese samples (support-ticket routing 40 / review sentiment 30 / comment spam 30 / support urgency 30). Protocol: temperature=0, one decision call per sample, no few-shot; CI cross-checked with statsmodels; per-sample records and all misjudgments are public in the repo.

Three findings you may find relevant:

1. Jev matches a much larger LLM judge at 4.5× lower latency. Jev (jev-1.13.0, native decisions API): 97.7% (127/130), ~890 ms, ¥0.105 / 1000 decisions at list price. GLM-5.3-flash: 97.7% (127/130) at ~4.0 s. DeepSeek-flash: 96.2% at ~1.1 s — and it missed 2 urgent support tickets, the highest-cost error type in routing scenarios.

2. Your Confidence-gated routing pattern holds empirically. All 3 Jev misjudgments scored below 0.7 confidence while low-confidence outputs were only 9.2% of runs — a 0.7 gate caught 100% of errors at a 9% escalation rate, on real borderline samples (2 of the 3 are also missed by GLM and DeepSeek).

3. Determinism: 3 independent runs produced zero judgment flips (127/130 each) — relevant for anyone building reproducible pipelines on the API.

Happy to turn any of this into a cookbook entry, demo, or benchmark contribution — tell us where it would be most useful.

[aoright]

These empirical results provide very valuable confirmation of Jev's behavioral characteristics on non-English classification tasks.

Notable Observations from Your Data

1. Zero Judgment Flips Across Runs:
   The finding that 3 independent runs yielded identical 127/130 outcomes illustrates a key architectural benefit of System One: by evaluating typed questions directly via next-token logit distributions without autoregressive sampling or reasoning trajectory divergence, execution remains completely deterministic at temperature 0.

2. Empirical Validation of the 0.70 Confidence Escalation Gate:
   Your observation that a 0.70 confidence threshold successfully intercepted 100% of misjudgments (3/3) while escalating only 9.2% of total traffic matches the intended operational design of confidence gating. In high-consequence routing, this delivers an optimal trade-off: over 90% of requests are resolved under sub-second latency at minimal token cost, while borderline edge cases are safely routed to higher-capacity generative fallbacks.

3. Cross-Lingual Consistency:
   While multilingual evaluations on long natural-language documents (e.g. Spanish in jev-acento) noted higher calibration error on long states, your findings indicate that bounded, domain-specific Chinese categorization tasks (routing, spam, sentiment, urgency) perform robustly with high precision.

4. Integration with the Jev Ecosystem:
   Your judgekit framework represents a clean, provider-agnostic runtime implementation for System One workflows. It is also an excellent candidate for the community ecosystem index (yibie/awesome-jev) under Evaluation & Benchmarking to help developers explore declarative YAML task evaluation.

[lexingtonhibiki]

Hi aoright,

Thanks for the careful read — glad the determinism and confidence-gate results were useful.

One accuracy note for the record: those numbers (127/130 with zero flips across 3 runs, and the 0.7 gate catching 3/3 misjudgments at 9.2% escalation) are on our in-house 130-item mini set.

We also froze a harder external set, where 120 spam items were labeled one by one by a human annotator; there Jev scores:

Metric	Score
Argmax accuracy	60.0%
Accuracy @ 0.10 probability threshold	68.3%

(in-sample, no holdout)

Both calibers, including the negative results and limitations, are documented in:

• README.md
• docs/jev-v4-report.md

---

Happy to be listed in awesome-jev — I've just opened a PR for judgekit under Evaluation & Benchmarking:
https://github.com/yibie/awesome-jev/PR/237

[aoright]

Thank you for sharing the additional context and referencing the harder 120-item annotated spam dataset and the detailed docs/jev-v4-report.md report.

Documenting empirical boundary conditions and probability-threshold recalibrations (such as the 60.0% argmax moving to 68.3% at tau=0.10) is exceptionally valuable for the community. It provides practitioners with realistic calibration targets when handling noisy, adversarial distributions compared to clean structured routing tasks.

Also, regarding awesome-jev: maintainer yibie merged your exact author description and {type: cli} metadata into the catalog on main. Looking forward to seeing further developments and benchmarks from the project!