Fix/benchmark

benchmarks/genui-formats/.gitignore

@@ -2,3 +2,5 @@
 node_modules
 results/raw/
 samples/
+vendor/openui-prompt-*.txt
+vendor/openui-schema-0.13.5.json

benchmarks/genui-formats/README.md

@@ -6,7 +6,7 @@ and **validated by its own validator**, across a ladder of models from flagship
 | Format | Wire format | Prompt source |
 | --- | --- | --- |
 | [MDMA](https://github.com/MobileReality/mr-mdma) | Markdown + YAML in ` ```mdma ` blocks | `buildSystemPrompt()` + per-model author variant |
-| [OpenUI Lang](https://github.com/thesysdev/openui) | line-oriented DSL | `benchmarks/system-prompt.txt` (their own published artifact) |
+| [OpenUI Lang](https://github.com/thesysdev/openui) | line-oriented DSL | `openuiLibrary.prompt()` from `@openuidev/react-ui@0.13.5` |
 | [json-render](https://github.com/vercel-labs/json-render) | JSONL of RFC-6902 patches | `catalog.prompt()` |
 | [AGenUI](https://github.com/AGenUI/AGenUI) | A2UI v0.9 JSON messages | `skills/a2ui-generation/SKILL.md` + required refs, flattened |
 | [A2UI](https://github.com/a2ui-project/a2ui) | A2UI v0.9 JSON messages (transport) | `a2ui_agent` Python prompt generator, standard v0.9 `basic` catalog |
@@ -77,6 +77,19 @@ vendor/               pinned upstream artifacts + PINS.txt (commit SHAs)
 results/              ledger, raw generations, results.json, REPORT.md
 ```
 
+## A2UI arm setup (one-off)
+
+The A2UI arm validates through A2UI's own Python SDK, which is not published to
+PyPI. Install it from the pinned commit before running or scoring that arm:
+
+```bash
+./a2ui-python/setup.sh          # clones the pinned commit, builds a venv
+```
+
+The checkout and venv default to `/tmp`, so the OS will eventually delete them;
+re-run the script when `pnpm verify` reports a missing interpreter. Set
+`A2UI_SRC`, `A2UI_VENV` and `A2UI_PYTHON` to keep them somewhere durable.
+
 ## Running
 
 ```bash

benchmarks/genui-formats/a2ui-python/setup.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Rebuild the Python environment the A2UI arm needs.
+#
+# The A2UI adapter shells out to a2ui_tool.py, which imports the A2UI project's
+# own SDK. That SDK is not on PyPI, so it is installed from a pinned commit of
+# a2ui-project/a2ui. Both the checkout and the venv live under /tmp by default,
+# which means they disappear whenever the OS cleans temporary files. Re-run this
+# script when `pnpm verify` reports a spawn failure for the A2UI adapter.
+#
+#   ./a2ui-python/setup.sh
+#
+# Override locations with A2UI_SRC / A2UI_VENV if you want them somewhere
+# durable, and set A2UI_PYTHON to the resulting interpreter so the adapter
+# finds it:
+#
+#   A2UI_SRC=~/.cache/a2ui-src A2UI_VENV=~/.cache/a2ui-venv ./a2ui-python/setup.sh
+#   export A2UI_PYTHON=~/.cache/a2ui-venv/bin/python
+
+set -euo pipefail
+
+# The commit the published results were generated against. Later commits added
+# an ANTLR build dependency that this arm does not need, so the pin is
+# deliberate rather than incidental.
+A2UI_COMMIT="${A2UI_COMMIT:-349c97909572c85111b6201c4e46169ae0277d9b}"
+A2UI_SRC="${A2UI_SRC:-/tmp/a2ui-src}"
+A2UI_VENV="${A2UI_VENV:-/tmp/a2ui-venv}"
+
+echo "A2UI commit : $A2UI_COMMIT"
+echo "checkout    : $A2UI_SRC"
+echo "venv        : $A2UI_VENV"
+
+rm -rf "$A2UI_SRC" "$A2UI_VENV"
+mkdir -p "$A2UI_SRC"
+
+# Shallow single-commit fetch: the full history is large and unnecessary.
+git -C "$A2UI_SRC" init -q
+git -C "$A2UI_SRC" remote add origin https://github.com/a2ui-project/a2ui.git
+git -C "$A2UI_SRC" fetch -q --depth 1 origin "$A2UI_COMMIT"
+git -C "$A2UI_SRC" checkout -q FETCH_HEAD
+
+# a2ui_agent's build reads ../../specification, so both packages install from
+# the repo layout rather than in isolation.
+python3 -m venv "$A2UI_VENV"
+"$A2UI_VENV/bin/pip" install -q \
+  "$A2UI_SRC/agent_sdks/python/a2ui_core" \
+  "$A2UI_SRC/agent_sdks/python/a2ui_agent"
+
+"$A2UI_VENV/bin/python" -c "import a2ui" && echo "OK: A2UI SDK installed"
+
+# The adapter also reads the standard v0.9 catalog straight from the checkout.
+test -f "$A2UI_SRC/specification/v0_9/catalogs/basic/catalog.json" \
+  && echo "OK: v0.9 basic catalog present"

benchmarks/genui-formats/package.json

@@ -20,6 +20,7 @@
     "@mobile-reality/mdma-prompt-pack": "workspace:*",
     "@mobile-reality/mdma-validator": "workspace:*",
     "@openuidev/lang-core": "0.2.10",
+    "@openuidev/react-ui": "0.13.5",
     "ajv": "^8.17.1",
     "dotenv": "^16.4.5",
     "openai": "^4.77.0",

benchmarks/genui-formats/results/REPORT.md

@@ -1,6 +1,6 @@
 # Generative-UI format reliability benchmark
 
-_Generated 2026-07-28T07:26:52.650Z · 1377 generations · 0 API errors_
+_Generated 2026-08-07T14:19:12.249Z · 1647 generations · 0 API errors_
 
 Five open-source generative-UI formats, each generated natively from its own published
 prompt and validated by its own validator, across a ladder of models.
@@ -21,9 +21,9 @@ They score very differently, which is why they are separate columns.
 
 | Rung | Model | Provider id | Generations |
 | --- | --- | --- | --- |
-| flagship | Opus 5 | `anthropic/claude-opus-5` | 450 |
-| mid | GPT-5.6-terra | `openai/gpt-5.6-terra` | 450 |
-| small | Gemma-4-26B-A4B | `google/gemma-4-26b-a4b-it` | 450 |
+| flagship | Opus 5 | `anthropic/claude-opus-5` | 540 |
+| mid | GPT-5.6-terra | `openai/gpt-5.6-terra` | 540 |
+| small | Gemma-4-26B-A4B | `google/gemma-4-26b-a4b-it` | 540 |
 
 All reached through OpenRouter, 5 repeats per scenario, temperature 0.7, max_tokens 8192 —
 identical settings for every format and model.
@@ -40,7 +40,7 @@ rendered — because a format that works four times in five is not something you
 | Format | Opus 5 (flagship) | Gemma-4-26B (open weights) | Drop |
 | --- | --- | --- | --- |
 | MDMA | 94.4% | 94.4% | 0.0pp |
-| OpenUI Lang | 83.3% | 55.6% | 27.8pp |
+| OpenUI Lang | 94.4% | 50.0% | 44.4pp |
 | json-render | 83.3% | 38.9% | 44.4pp |
 | A2UI (transport) | 83.3% | 44.4% | 38.9pp |
 | AGenUI (A2UI SDK) | 38.9% \* | 77.8% | N/A |
@@ -91,8 +91,8 @@ Prompt size matters here — you pay it on every request:
 
 | Format | Prompt tokens ↓ |
 | --- | --- |
-| OpenUI Lang | 5172 |
 | MDMA | 5910 |
+| OpenUI Lang | 6978 |
 | json-render | 8466 |
 | A2UI (transport) | 16085 |
 | AGenUI (A2UI SDK) | 19689 |
@@ -125,7 +125,7 @@ Every metric below is marked **↑ higher is better** or **↓ lower is better**
 | Formats | 5 |
 | Models | 3 |
 | Repeats per scenario | 5 |
-| Total generations | 1377 |
+| Total generations | 1647 |
 | Temperature | 0.7, identical for every format and model |
 | Max tokens | 8192, identical |
 
@@ -137,7 +137,7 @@ artifact or prompt-generation API.
 | Format | Source | Prompt tokens (↓ lower is better) |
 | --- | --- | --- |
 | MDMA | buildSystemPrompt() + per-model author variant (packages/prompt-pack) | 5910 |
-| OpenUI Lang | vendor/openui-system-prompt.txt (thesysdev/openui @ 65b5f93) | 5172 |
+| OpenUI Lang | openuiLibrary.prompt() from @openuidev/react-ui@0.13.5 | 6978 |
 | json-render | catalog.prompt() — @json-render/core 0.19.0, shadcn catalog + Chart | 8466 |
 | A2UI (transport) | a2ui_agent TransportFormat.prompt_generator.generate() — A2UI's own Python SDK | 16085 |
 | AGenUI (A2UI SDK) | flattened AGenUI skills/a2ui-generation SKILL.md + refs (AGenUI @ 3e79bea) | 19689 |
@@ -149,11 +149,11 @@ Share of generations that parse and validate — i.e. that a renderer could rend
 | Model | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
 | **flagship** |  |  |  |  |  |
-| Opus 5 | 100.0% | 98.8% | 100.0% | 96.4% | 100.0% |
+| Opus 5 | 100.0% | 100.0% | 100.0% | 96.4% | 100.0% |
 | **mid** |  |  |  |  |  |
-| GPT-5.6-terra | 91.1% | 100.0% | 93.3% | 86.7% | 81.1% |
+| GPT-5.6-terra | 91.1% | 98.9% | 93.3% | 86.7% | 81.1% |
 | **small** |  |  |  |  |  |
-| Gemma-4-26B-A4B | 98.9% | 83.3% | 83.3% | 81.1% | 93.3% |
+| Gemma-4-26B-A4B | 98.9% | 78.9% | 83.3% | 81.1% | 93.3% |
 
 > ⚠️ **AGenUI's column is measured on a looser standard than AGenUI's own tooling applies.** Every
 > format here is checked for structural renderability, but AGenUI additionally ships an 889-line
@@ -173,7 +173,7 @@ It is still a real cost. Truncation here means: at 8k output tokens, that format
 | Model | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
 | **flagship** |  |  |  |  |  |
-| Opus 5 | 4.4% | 8.9% | 12.2% | 7.8% | 52.2% |
+| Opus 5 | 4.4% | 5.6% | 12.2% | 7.8% | 52.2% |
 | **mid** |  |  |  |  |  |
 | GPT-5.6-terra | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
 | **small** |  |  |  |  |  |
@@ -187,11 +187,11 @@ you are shipping a product: a format that works 4 times in 5 still breaks in pro
 | Model | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
 | **flagship** |  |  |  |  |  |
-| Opus 5 | 94.4% | 83.3% | 83.3% | 83.3% | 38.9% |
+| Opus 5 | 94.4% | 94.4% | 83.3% | 83.3% | 38.9% |
 | **mid** |  |  |  |  |  |
-| GPT-5.6-terra | 83.3% | 100.0% | 72.2% | 61.1% | 72.2% |
+| GPT-5.6-terra | 83.3% | 94.4% | 72.2% | 61.1% | 72.2% |
 | **small** |  |  |  |  |  |
-| Gemma-4-26B-A4B | 94.4% | 55.6% | 38.9% | 44.4% | 77.8% |
+| Gemma-4-26B-A4B | 94.4% | 50.0% | 38.9% | 44.4% | 77.8% |
 
 ## 3. Shape stability — ↑ higher is better (diagnostic only — NOT comparable across formats)
 
@@ -212,7 +212,7 @@ Share of scenarios where all 5 repeats produced the same component structure.
 | **flagship** |  |  |  |  |  |
 | Opus 5 | 16.7% | 5.6% | 5.6% | 5.6% | 5.6% |
 | **mid** |  |  |  |  |  |
-| GPT-5.6-terra | 72.2% | 5.6% | 5.6% | 11.1% | 5.6% |
+| GPT-5.6-terra | 72.2% | 22.2% | 5.6% | 11.1% | 5.6% |
 | **small** |  |  |  |  |  |
 | Gemma-4-26B-A4B | 72.2% | 5.6% | 0.0% | 5.6% | 5.6% |
 
@@ -226,29 +226,29 @@ Cheap output nobody can render is not cheap.
 | Model | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
 | **flagship** |  |  |  |  |  |
-| Opus 5 | 1666 | 1814 | 4019 | 3892 | 7209 |
+| Opus 5 | 1666 | 2125 | 4019 | 3892 | 7209 |
 | **mid** |  |  |  |  |  |
-| GPT-5.6-terra | 554 | 567 | 1020 | 1081 | 1674 |
+| GPT-5.6-terra | 554 | 624 | 1020 | 1081 | 1674 |
 | **small** |  |  |  |  |  |
-| Gemma-4-26B-A4B | 448 | 417 | 836 | 1300 | 1485 |
+| Gemma-4-26B-A4B | 448 | 525 | 836 | 1300 | 1485 |
 
 **Efficiency — ↑ higher is better (renderable output per 1k output tokens):**
 
 | Model | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
 | **flagship** |  |  |  |  |  |
-| Opus 5 | 0.60 | 0.54 | 0.25 | 0.25 | 0.14 |
+| Opus 5 | 0.60 | 0.47 | 0.25 | 0.25 | 0.14 |
 | **mid** |  |  |  |  |  |
-| GPT-5.6-terra | 1.64 | 1.77 | 0.91 | 0.80 | 0.48 |
+| GPT-5.6-terra | 1.64 | 1.59 | 0.91 | 0.80 | 0.48 |
 | **small** |  |  |  |  |  |
-| Gemma-4-26B-A4B | 2.21 | 2.00 | 1.00 | 0.62 | 0.63 |
+| Gemma-4-26B-A4B | 2.21 | 1.50 | 1.00 | 0.62 | 0.63 |
 
 ## 5. Failure taxonomy — ↓ lower is better (0 is perfect)
 
 | Format | broken-reference | no-structured-output | off-task | parse-error | prose-leakage | schema-error | truncated | unknown-component |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | MDMA | 0 | 6 | 2 | 1 | 0 | 0 | 4 | 0 |
-| OpenUI Lang | 142 | 0 | 0 | 0 | 0 | 57 | 8 | 1 |
+| OpenUI Lang | 38 | 0 | 0 | 0 | 0 | 55 | 7 | 5 |
 | json-render | 27 | 0 | 0 | 0 | 0 | 26 | 11 | 0 |
 | A2UI (transport) | 0 | 0 | 0 | 32 | 0 | 0 | 7 | 0 |
 | AGenUI (A2UI SDK) | 3 | 13 | 7 | 0 | 2 | 0 | 47 | 0 |
@@ -257,19 +257,19 @@ Cheap output nobody can render is not cheap.
 
 | Family | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
-| contact-form | 100.0% | 100.0% | 93.3% | 82.2% | 77.8% |
-| data-table | 82.2% | 86.7% | 82.2% | 66.7% | 66.7% |
-| chart | 97.8% | 93.3% | 95.6% | 95.6% | 80.0% |
-| callout | 100.0% | 91.1% | 95.6% | 95.6% | 93.3% |
+| contact-form | 100.0% | 91.1% | 93.3% | 82.2% | 77.8% |
+| data-table | 82.2% | 82.2% | 82.2% | 66.7% | 66.7% |
+| chart | 97.8% | 91.1% | 95.6% | 95.6% | 80.0% |
+| callout | 100.0% | 97.8% | 95.6% | 95.6% | 93.3% |
 | button | 100.0% | 97.8% | 88.9% | 100.0% | 88.9% |
-| tasklist | 91.1% | 77.8% | 73.3% | 73.3% | 37.8% |
+| tasklist | 91.1% | 84.4% | 73.3% | 73.3% | 37.8% |
 
 **By variant** — the adversarial variant is where formats separate:
 
 | Variant | MDMA | OpenUI Lang | json-render | A2UI (transport) | AGenUI (A2UI SDK) |
 | --- | --- | --- | --- | --- | --- |
-| minimal | 100.0% | 93.3% | 95.6% | 92.2% | 87.8% |
-| realistic | 98.9% | 97.8% | 90.0% | 87.8% | 71.1% |
+| minimal | 100.0% | 96.7% | 95.6% | 92.2% | 87.8% |
+| realistic | 98.9% | 93.3% | 90.0% | 87.8% | 71.1% |
 | adversarial | 86.7% | 82.2% | 78.9% | 76.7% | 63.3% |
 
 ## 7. Does adding few-shot examples help a weak model? — no
@@ -289,6 +289,41 @@ asymmetry disadvantaged A2UI, we re-ran the same 18 scenarios x 5 repeats on Gem
 `<a2ui-json>` block at all — the model answered in prose — and parse errors rose from 17 to
 27. At this model size the instruction gets lost in the context rather than reinforced.
 
+## 8. OpenUI's published prompt is stale against the library it ships
+
+thesysdev publish `benchmarks/system-prompt.txt` and use it in their own benchmark. It is
+byte-identical to their repository today, but it no longer matches the library they ship on
+npm: `openuiLibrary.prompt()` from `@openuidev/react-ui@0.13.5` produces a different prompt.
+
+|  | Published artifact | Shipped library v0.13.5 |
+| --- | --- | --- |
+| Size | 13,080 chars | 17,431 chars |
+| `## Examples` | present | **removed** |
+| `## Action - Button Behavior` | absent | **added** |
+| `## Final Verification` | absent | **added** |
+| Components | base set | **+Modal, +Action** |
+
+Anyone installing the package today gets the second one, so **the scored OpenUI arm uses the
+shipped library prompt**, with the schema regenerated from the same version so parser and
+prompt agree. We re-ran all 270 generations rather than adjusting anything after the fact.
+
+The superseded run is kept here because the comparison is informative:
+
+| Model | Published artifact | Shipped library | Delta |
+| --- | --- | --- | --- |
+| Opus 5 | 83.3% | 94.4% | +11.1pp |
+| GPT-5.6-terra | 100.0% | 94.4% | -5.6pp |
+| Gemma-4-26B | 55.6% | 50.0% | -5.6pp |
+
+"Every time" rate, five repeats per scenario. The correction gained 2 scenarios on the
+flagship and lost 1 on each cheaper model, averaging out to no change across the ladder.
+Correcting it was therefore about accuracy, not about advantage: the published artifact was
+not systematically unfair to OpenUI, and scored better on two rungs of three.
+
+The direction is consistent with section 7. The shipped prompt trades worked examples for
+prose rules; a strong model applies rules, a weak one needs the examples. On Gemma the
+largest failure category became `missing-required`, which is what losing examples predicts.
+
 ## Method and fairness
 
 - Every format uses **its own published prompt, unmodified**, from its own official artifact