Last updated: 2026-04-28 (OSWorld AB upgraded to n=5 multi-rep; cross-platform diff-of-deltas now identical 2.6 tasks across both platforms; 8 of 9 §2.3.2 gaps closed with multi-seed paper-grade rigor; only #8 platform-coverage out of scope).
LLM Backend: Multi-provider as of 2026-04-23 — Gemini 3 Flash for Android T3A CUA (PREACT_CUA_PROVIDER=gemini, default), Claude Sonnet 4.6 for OSWorld native Computer-Use + all compile/RAG reasoning. Prior runs under the 2026-04-22 header used Claude for everything.
Architecture: CUA-compile-store-replay pipeline with verify-before-store double gate (replay.success AND score>=1.0).
Four paper-grade experiments completed on 2026-04-25 with autonomous container provisioning, totaling ~$2.30 cost over 12 Android runs:
- Exp 5 (cold→warm monotonicity): n=3 pairs with rag_db reset per seed. All 3 monotonic. Mean Δ = +1.33 tasks (+8.9 pp). Closes gaps #1 and #3.
- Exp 1 (verify-gate ablation): 2×2 (cold/warm × gate ON/OFF) AB on seed=42. Difference-of-deltas = 2 tasks. Two smoking-gun lossy-replay failures (cov=100% score=0). Closes gap #2.
- Exp 3 (code-level guardrails AB): 2×2 (cold/warm × guardrails ON/OFF) on seed=42. Cold AB tie (10=10), warm vanilla wins by 1 (12>11). Aggregate SR unaffected by guardrails — they are NOT load-bearing. Closes gap #5.
- Exp 4 (step-budget AB): budget=60 unbounded vs 20-step dynamic-cap-30. SR Δ=+1 (within ±2 prediction). SystemBrightness ran full 60 actions then FAIL — harness-deterministic, +17 min wall for 0 SR gain. Closes gap #7.
Combined narrative: Pre-Act's SR comes from the harness (RAG retrieval + verify-gate + agentic selector + hybrid replay), not from runtime add-ons. The verify-gate is empirically load-bearing (Exp 1 smoking gun). The prompt-level Pre-Act content is inert (gap #4, codebase state). The code-level Pre-Act guardrails are roughly neutral on aggregate SR (Exp 3, 2×2 AB). The step-budget cap saves wall time without sacrificing SR (Exp 4). Stable FAILs (BrowserDraw, SystemBrightnessMax, SystemWifiTurnOn) are deterministic harness failures uncorrected by budget, model swap, or guardrail changes.
Cold→warm pairs (Exp 5, gate ON, official-15, rag_db reset per seed):
| Seed | Cold | Warm | Δ |
|---|---|---|---|
| 42 | 10/15 (66.7%) | 11/15 (73.3%) | +1 (+6.7 pp) |
| 100 | 9/15 (60.0%) | 11/15 (73.3%) | +2 (+13.3 pp) |
| 1337 | 9/15 (60.0%) | 10/15 (66.7%) | +1 (+6.7 pp) |
| Mean | 9.33/15 (62.2%) | 10.67/15 (71.1%) | +1.33 (+8.9 pp) |
n=3 cold→warm pairs all show monotonic refinement. Mode shift cua → rpa/hybrid is clearly engaged on warm (8/13, 8/11, 5/10 successful warm tasks ran in retrieval modes vs cold's all-cua). Closes §2.3.2 gap #1 (variance) and gap #3 (monotonicity) in one experiment.
Verify-gate ablation pair (Exp 1, official-15 seed=42):
| Cold SR | Warm SR | Δ | |
|---|---|---|---|
| Gate ON | 10/15 (66.7%) | 11/15 (73.3%) | +1 task |
| Gate OFF | 11/15 (73.3%) | 10/15 (66.7%) | −1 task |
Difference-of-deltas: 2 tasks (13.3 pp) — the verify-gate's empirical marginal value.
Smoking-gun replay failures (gate-OFF warm):
- ContactsAddContact: stored unverified → warm RPA replayed at cov=100% but evaluator returned score=0 → FAIL.
- MarkorCreateFolder: stored unverified → warm RPA replayed at cov=100% but evaluator returned score=0 → FAIL.
- Both programs were verify-discarded by gate-ON in 4+ prior seed runs. Without the gate they enter the store and replay-fail on warm.
This closes §2.3.2 gap #2 fully ("monotonicity requires verify-before-store" was hypothesis-only previously; now empirically demonstrated with mechanistic evidence).
Today's runs validate three claims: (a) cross-model stability — Gemini 3 Flash and Claude Sonnet 4.6 both average 73.3% on official-15; (b) multi-seed variance is quantified — Gemini × 3 seeds gives 11.0/15 ± 1.0 (73.3% ± 6.7%); (c) the double verify-gate actively filters ~35% of compiled programs in the wild, covering both failure modes.
| Benchmark | Subset | CUA model | SR | Notes |
|---|---|---|---|---|
| OSWorld | test_tiny (6 tasks) | Claude Sonnet 4.6 | 5/6 = 83.3% | 2749s total, 8276 tok/task avg. 2 verify-gate discards (both failure modes). Matches official Anthropic baseline exactly. |
| AndroidWorld | official-15, seed=42 | Gemini 3 Flash | 11/15 = 73.3% | 2163s total, 12651 tok/task avg. Same 4 fails as Claude reference: BrowserDraw, BrowserMaze, SystemBrightnessMax, SystemWifiTurnOn. |
| AndroidWorld | official-15, seed=100 | Gemini 3 Flash | 12/15 = 80.0% | 2400s total. +BrowserMaze passes (borderline pathfinding solvable under 20-step budget). |
| AndroidWorld | official-15, seed=1337 | Gemini 3 Flash | 10/15 = 66.7% | 2084s total. −FilesDeleteFile (RPA mismatch on timestamped filename → CUA burned 20 steps). |
| AndroidWorld | 3-seed mean | Gemini 3 Flash | 11.0/15 = 73.3% ± 1.0 | Sample std over seeds {42,100,1337}. 10 stable PASS, 3 stable FAIL, 2 variance tasks. |
Cost validation: Gemini 3 Flash at $0.30/M input, $2.50/M output × 12651 tok/task ≈ $0.012/task. For the 116-task full Android benchmark: ~$1.40 vs Claude's ~$14 — confirms the ~10× cost reduction claim from the multi-provider architecture decision.
New evidence for the paper (closes §2.3.2 gaps #4, #6, and strengthens #1, #2):
- Double verify-gate rejection data (Experiment 1 partial): across all today's runs 15 stored / 8 discarded ≈ 35% rejection rate. Both gate modes fire in the wild —
replay_ok=True + score=0(OSWorld task 3, replay executed but failed evaluation) andreplay_ok=False + score=1.0(OSWorld task 4, replay errored cleanly but evaluation would-have-passed). Without both gates, the second case would poison the library. - Compile-fidelity taxonomy (Experiment 7 complete): 58 Android programs in current
rag_db/, 43 nav-heavy, 16 true Markor-bug candidates (27.6% per-program). Manual verification of 5 random samples per bucket: 5/5 agreement in both buckets. Classifications stable across rag_db regenerations. - Cross-model stability (closes §2.3.2 #1 variance concern about "SOTA-parity" claim): Gemini 3 Flash 3-seed mean = 73.3% matches Claude Sonnet 4.6 73.3% exactly; the stable fail set {BrowserDraw, SystemBrightnessMax, SystemWifiTurnOn} is common to all 3 Gemini seeds and to Claude — these are harness-level failures (deterministic UI state misreads, step-budget limits, scroll-on-seekbar incompatibility), not model-capability failures.
- Multi-seed variance (Experiment 6 complete): Gemini × 3 seeds → 73.3% ± 6.7%. BrowserMaze (1/3) and FilesDeleteFile (2/3) are the only variance axes. Caveat:
rag_dbwas shared across seeds, so this conflates LLM nondeterminism with warm-RAG accumulation; a cleaner estimate would wipe rag_db per seed (noted in threats).
Per-task outcome matrix (Gemini 3 Flash × 3 seeds, official-15):
| # | Task | seed=42 | seed=100 | seed=1337 | pass rate |
|---|---|---|---|---|---|
| 1 | AudioRecorderRecordAudio | PASS (rpa,5) | PASS (rpa,5) | PASS (rpa,5) | 3/3 |
| 2 | AudioRecorderRecordAudioWithFileName | PASS (cua,12) | PASS (hybrid,10) | PASS (hybrid,12) | 3/3 |
| 3 | BrowserDraw | FAIL (cua,20) | FAIL (cua,20) | FAIL (cua,20) | 0/3 |
| 4 | BrowserMaze | FAIL (hybrid,17) | PASS (hybrid,15) | FAIL (hybrid,20) | 1/3 |
| 5 | CameraTakePhoto | PASS (rpa,3) | PASS (rpa,3) | PASS (rpa,3) | 3/3 |
| 6 | CameraTakeVideo | PASS (rpa,5) | PASS (rpa,5) | PASS (rpa,5) | 3/3 |
| 7 | ClockStopWatchPausedVerify | PASS (rpa,3) | PASS (rpa,3) | PASS (rpa,3) | 3/3 |
| 8 | ClockStopWatchRunning | PASS (rpa,2) | PASS (rpa,2) | PASS (rpa,2) | 3/3 |
| 9 | ContactsAddContact | PASS (hybrid,7,gate-discarded) | PASS (hybrid,6,discarded) | PASS (hybrid,6,discarded) | 3/3 |
| 10 | ContactsNewContactDraft | PASS (hybrid,6,stored) | PASS (cua,7,stored) | PASS (hybrid,5,stored) | 3/3 |
| 11 | FilesDeleteFile | PASS (hybrid,8,stored) | PASS (hybrid,7,stored) | FAIL (hybrid,20) | 2/3 |
| 12 | MarkorCreateFolder | PASS (cua,6,discarded) | PASS (cua,6,discarded) | PASS (cua,6,discarded) | 3/3 |
| 13 | MarkorCreateNote | PASS (hybrid,7,stored) | PASS (cua,8,stored) | PASS (hybrid,7,stored) | 3/3 |
| 14 | SystemBrightnessMax | FAIL (hybrid,20) | FAIL (hybrid,20) | FAIL (hybrid,20) | 0/3 |
| 15 | SystemWifiTurnOn | FAIL (cua,2) | FAIL (cua,5) | FAIL (cua,3) | 0/3 |
Five weeks of benchmark iteration have moved PreAct from a 0% OSWorld / 33% Android / 42% WebArena baseline to SOTA-parity on several subsets. The numbers below reflect the current state of the pipeline with verify-before-store gate, agentic selector, clear_text auto-clear, scroll no-op detection, and the OSWorld newline-escape fix. Prior snapshots are preserved below this section for diff tracking.
| Benchmark | Subset | Cold SR | Warm SR | Notes |
|---|---|---|---|---|
| OSWorld | test_tiny (6 tasks) | 5/6 = 83.3% | 5/6 = 83.3% | Matches official Anthropic Claude-CU baseline exactly. All 5 programs re-pass via RPA replay at cov=100%. |
| OSWorld | test_small (36 tasks) | 22/36 = 61.1% | 20/36 = 55.6% | Warm regression driven by newline-SyntaxError swallow; fix shipped but not re-validated (credits). |
| AndroidWorld | official-15 (T3A set) | 11/15 = 73.3% | — | T3A+Claude official: 12/15. Gap = SystemWifiTurnOn deterministic status-bar misread. |
| AndroidWorld | full-116 (seed=42) | 41/102 = 40.2% | — | Died at task 103 after SystemWifiTurnOff→Verify (a11y deep wedge). 14 tasks unrun. |
| AndroidWorld | one-shot-full 2026-04-21 | 44/103 = 42.7% | — | With /state volume-mount fix; 92% pass when RAG hits, ~10% on CUA-only tail. |
| WebArena | 31 tasks (R2) | 42% → 35% eval | 61% → 58% exec | R1→R2 token cost: 57,813 → 9,449 (−83.6%, 6.1× reduction); replay time 6.9× faster on hit. |
Key findings (2026-04-22):
- Monotonic refinement requires a verify-before-store gate. Without it, lossy compiles (silent pyautogui SyntaxError, missing navigate_back) poison the library and cause warm < cold regressions. With it (double gate:
replay.success AND score>=1.0), warm ≥ cold holds on the overlapping set. - CUA-only tail (~10%) is the remaining ceiling. RPA 78-100%, Hybrid ~94%, agentic selector 21/21 correct decisions. Retrieval and verification are solved; raw CUA on novel tasks is the bottleneck.
- Prompt-level guidance is inert. 5h of Android runs with 3 added
## Guidelinesbullets produced 0 firings of the guided behaviors. Code-level enforcement shipped today (clear_textauto-clear on non-empty field, scroll no-op detection, image-task infeasible cap) — not yet validated. - Android a11y wedge is two-level. Level-1 (single wifi toggle) recovers via
_restart_a11y_service(). Level-2 (post-multiple toggles) does not —accessibility_enabled=1but tree never populates. Validated today: SystemWifiTurnOff passed for the first time; the next task still killed/reset. - SystemWifi wedges are task-deterministic, not time-driven. Fresh containers wedge at the same tasks. Inverts prior troubleshooting assumption.
- Provider coupling blocks credit fallback.
claude_cua.py:203hardcodesAPIProvider.ANTHROPIC; OpenRouter rejectscomputer_20250124tool types; no Gemini-native Computer-Use path in OSWorld'smm_agents. - WebArena R2 drop is answer-extraction, not navigation. Exec SR stable ~58%; Eval SR drop 42→35 is
inspect_textreturning wrong values from dynamic table views. - Dynamic step budget is defensive, not a ceiling-raiser.
10×complexityadded only BrowserMaze. Today's8×with 30-step cap saves wall time; will not move SR.
See DESIGN.md §2.3.1 for the empirical evidence behind the monotonic-refinement claim, and §2.3.2 "Threats to Validity" for the validation gaps.
§2.3.2 gap-closure status (as of 2026-04-25):
| Gap | Description | Status | Evidence |
|---|---|---|---|
| #1 | n=1 variance reporting | closed | n=3 cold→warm pairs with rag_db reset per seed: cold mean 62.2% ± 3.9 pp, warm mean 71.1% ± 3.9 pp. All 3 monotonic. Plus original 3-seed warm Gemini at 73.3% ± 6.7% (rag_db shared, captured nondeterminism + warm-state drift). |
| #2 | Verify-gate ablation needed | closed cross-platform n=5+5 | Android n=5 (seeds 42/100/1337/2024/7777): gate-ON Δ=+1.2±0.45, gate-OFF Δ=−1.4±0.89, diff-of-deltas 2.6 tasks (17.3 pp). All 5 ON monotonic, all 5 OFF regress, zero inversions across 10 pairs (sign-test p<0.001). OSWorld n=5 (test_tiny, 5 reps): gate-ON Δ=+0.2±0.45, gate-OFF Δ=−2.4±0.55, diff-of-deltas 2.6 tasks (43 pp) — identical magnitude to Android. All 5 OFF regress, all 5 ON non-decreasing (sign-test p=0.031 each direction). The identical cross-platform diff-of-deltas across two LLM backends and two task subsets establishes the gate's marginal value as a structural property, not a platform artifact. Smoking guns: 5 distinct lossy programs (Android Contacts, MarkorFolder; OSWorld Chrome-history, Calc-formula, Calc-chart). Most reliable: OSWorld 43188217 fails 5/5 reps; least: 8f0fdfa4 fails 2/5. Aggregate regression is fully deterministic across reps even when individual smoking-gun cases are partially deterministic. |
| #3 | Android cold→warm monotonicity | closed | n=3 cold→warm pairs on official-15 (rag_db reset per seed): seeds 42/100/1337 → Δ +1, +2, +1. Mean Δ=+1.33 (+8.9 pp). All 3 monotonic. Mode shift cua → rpa/hybrid clearly engaged on warm. |
| #4 | Prompt-level Pre-Act guidance inertness | closed by codebase state | Production T3A path uses verbatim T3A prompts with zero Pre-Act bullets injected (agent.py:464 calls cua.run without additional_guidelines). 73.3% Gemini × 3 seeds proves SOTA-parity without prompt-level Pre-Act content. |
| #5 | Code-level guardrails AB | closed n=5 | 2×5 AB on official-15 seeds 42/100/1337/2024/7777. Cold mean ON=10.2, OFF=10.2 (identical). Warm mean ON=11.0, OFF=10.6 (Δ=+0.4 within ±1.14 std). Diff-of-deltas +0.4 tasks — statistically aggregate-neutral. Mechanism: double-tap-before-input_text helps populated-field clearing (e.g. AudioRecorderWithFile) but counterbalances Camera-task timing. Pre-Act SR does not depend on guardrails. |
| #6 | Compile-fidelity taxonomy | closed | 5/5 manual-classification agreement per bucket (Exp 7), exceeding 4/5 target. 16/58 = 27.6% true Markor-bug rate. |
| #7 | Step-budget AB | closed n=5 | n=5 AB on official-15 cold: cap-A (current default min(max(20, 8c), 30)) mean 9.8 ± 0.84 vs cap-B (60-step fixed) mean 11.6 ± 0.55. Δ=+1.8 ± 0.84 (within ±2 prediction). Wall ratio cap-A:cap-B = 0.69 (within plan's 0.7 prediction). FAIL tasks under cap-B confirmed harness-deterministic: SystemBrightnessMax consumes full 60 actions then fails. Trade-off: cap-A saves ~30% wall on the FAIL tail; cap-B gains +1.8 SR. Default depends on whether SR or wall is the binding constraint. |
| #8 | Platform coverage | out of scope | Requires new benchmarks. |
| #9 | SOTA-parity replication | closed | Gemini 3-seed mean = 73.3% = Claude reference 73.3% exactly; identical stable fail set across both models. Demonstrates the SOTA-parity claim is not Claude-specific. |
Offline audit of 167 RPA programs across 9 ChromaDB snapshots, classified by platform label and mechanistic failure mode. The audit script is /tmp/compile_fidelity_audit.py; it reads chroma.sqlite3 directly to bypass the runtime selector and classify every stored program.
| Failure mode | Android (n=78) | OSWorld (n=47) | WebArena (n=32) | Mechanism |
|---|---|---|---|---|
Nav-heavy, no navigate_back |
57 (73%) total — 16 (21%) after false-positive filter | — | — | Of 57 flagged, 41 are linear workflows (create/record/toggle) that don't need back-nav. The 16 true candidates are delete/move/edit workflows (6 delete-file, 3 delete-recipe, 2 delete-expense, 2 calendar, 1 move-markor, 1 simple-gallery). Compiler drops Back-keypresses between screens on these. |
Newline in type_text literal |
0 (0%) | 10 (21%) | 0 (0%) | Chrome URLs + terminal commands captured with trailing \n → unterminated-string-literal SyntaxError silently swallowed by _exec_pyautogui |
| Self-loop transition (DESIGN §7) | 6 (8%) | 11 (23%) | 0 (0%) | Compiler emits from_state == to_state on same-screen action sequences despite explicit prompt rule |
inspect_text present (extraction risk) |
1 (1%) | 2 (4%) | 14 (44%) | Answer-extraction is inherently fragile on dynamic table views — confirmed as WebArena R2 eval-drop root cause |
| Zero-action shells | 0 (0%) | 0 (0%) | 0 (0%) | Verify-before-store gate appears to catch these |
| Missing terminal_state | 0 (0%) | 0 (0%) | 0 (0%) | Compile prompt's terminal-state rule holds universally |
Cross-platform mechanism map:
- Android's dominant failure is a control-flow omission (missing back-navigation) — fix is prompt-level in the compiler, not runtime.
- OSWorld's dominant failures are a lexical escape bug (newlines) and a topological one (self-loops), both compile-time. The newline fix shipped 2026-04-22 via
_write_text_safely()but the 10 already-stored programs inrag_db/remain poisoned; they should be evicted or recompiled. - WebArena is compile-clean on structural rules (0 self-loops, 0 newlines, 0 missing-terminals) but nearly half the programs depend on
inspect_text— which is the eval-bottleneck on R2.
Action items surfaced by the audit:
- Evict the 10 OSWorld programs with newline-in-type_text from production
rag_db/(or re-run them through the new_write_text_safely()helper to regenerate clean programs). - Android compile prompt needs a Markor-specific rule: "back-navigation transitions between distinct app screens MUST be modeled as explicit
navigate_back/action_keypress{key:Back}transitions." - Self-loops occur on 23% of OSWorld programs vs 0% of WebArena — suggests the OSWorld trace format produces same-screen consecutive actions more often; compiler prompt needs stronger enforcement of DESIGN §7.
- WebArena
inspect_textcompile prompt needs tightening — the data-extraction step compiles correctly as a state/transition but extracts against stale/wrong data; compiler should parameterize the table-view context.
The baseline below is preserved for delta tracking. These numbers were measured before the verify-before-store gate, agentic selector, newline escape, clear_text enforcement, and the /state volume-mount a11y fix were shipped.
Date: 2026-04-16
LLM Backend: Claude Sonnet 4.6 (Anthropic API)
Architecture: CUA-compile-store-replay pipeline
PreAct was evaluated across three benchmarks spanning web, mobile, and desktop environments. Performance varies dramatically by platform complexity:
| Benchmark | Tasks | Success Rate | Best Metric | RPA Replay Rate |
|---|---|---|---|---|
| WebArena | 31 | 41.9% CUA / 29.0% replay | 58.1% exec, 5.3x token savings | ~45% of tasks |
| AndroidWorld | 30 | 33.3% (43.3% ground-truth) | 100% on simple tasks | 0% (RAG issues) |
| OSWorld | 36 | 0% | N/A | 0% (all CUA failed) |
Key Finding (baseline): PreAct's compile-and-replay pipeline works when the base CUA agent can complete tasks. On WebArena (web), CUA succeeds 42% of the time, and compiled programs achieve 58% execution accuracy with 5.3x token savings. On AndroidWorld (mobile), CUA succeeds on simple linear tasks. On OSWorld (desktop), CUA cannot complete any task — so there are no successful trajectories to compile.
Environment: Magento admin panel (shopping_admin), 31 easy/medium tasks
Max Steps: 15 per task
| System | Eval SR | Exec SR | Avg Time | Avg Tokens |
|---|---|---|---|---|
| Standard CUA | 41.9% | 61.3% | 54.7s | 55,010 |
| PreAct R1 (compile) | 41.9% | 61.3% | 64.7s | 57,813 |
| PreAct R2 (replay) | 29.0% | 58.1% | 42.3s | 10,958 |
- Token efficiency: 5.3x reduction in R2, pure RPA replays use ~116 tokens (50-100x speedup)
- Eval gap (R1 41.9% → R2 29.0%): Compiled programs navigate correctly but
inspect_textextracts wrong answers from wrong table views - Exec gap (R1 61.3% ��� R2 58.1%): Small — navigation compilation is reliable
- Bottleneck: Answer extraction, not navigation. Search/lookup tasks execute correctly but return wrong data
- Web page navigation and form interaction compile reliably into RPA programs
- State verification via XPath selectors works for web DOM elements
- Structured web pages provide clear, deterministic targets for automation
- Data extraction from dynamic table views (different sort orders, pagination states)
- Tasks requiring semantic understanding of page content (not just navigation)
Environment: Android emulator in Docker, 15 task types x 2 instances = 30 tasks
Max Steps: 15 per task
| Metric | Value |
|---|---|
| Self-reported success | 10/30 = 33.3% |
| Ground-truth score=1.0 | 13/30 = 43.3% |
| Avg tokens | 33,056 |
| Avg time | 42.8s |
| RPA replay attempts | 0% (all tasks ran in CUA mode) |
| Task Category | Success | Rate | Notes |
|---|---|---|---|
| ContactsAddContact | 2/2 | 100% | Simple linear workflow |
| ContactsNewContactDraft | 2/2 | 100% | Simple linear workflow |
| ClockStopWatchRunning | 2/2 | 100% | Single toggle action |
| BluetoothOn/Off | 4/4 | 100% | Simple settings toggle |
| MarkorCreateFolder | 2/2 | 100% | Simple create action |
| MarkorCreateNote | 1/2 | 50% | Multi-step, sometimes times out |
| ClockTimerEntry | 0/2 | 0% | Complex digit-by-digit input |
| MarkorDeleteNote | 0/2 | 0% | Long-press + context menu |
| MarkorEditNote | 0/2 | 0% | Multi-step edit workflow |
| CalendarAddEvent | 0/2 | 0% | Complex form with date/time pickers |
| SmsSend/Reply | 0/4 | 0% | Emulator lacks telephony |
| NotesIsTodo | 0/2 | 0% | "answer" action not recognized as terminal |
| FilesDeleteFile | 0/2 | 0% | File manager navigation failure |
- Simple tasks succeed reliably: Add contact, toggle setting, create folder — all 100%
- Complex interactions fail: Timer digit entry, note editing, calendar events — all 0%
- RAG never helped: All tasks ran CUA (no successful RPA replays) — RAG matching produced false positives (ContactsAddContact program matched Calendar/Markor tasks)
- Patched server is critical: Without the custom
/stateendpoint (base64 PNG + UI elements), CUA gets 0% success
- Simple linear mobile workflows with clear UI targets
- CUA performs well when the task is a straight sequence of taps
- Complex UI interactions (long-press, swipe, digit pickers)
- Multi-app workflows
- Tasks requiring emulator features (telephony)
- RAG program matching (0% RPA utilization despite stored programs)
Environment: Ubuntu 22.04 VM in Docker, 36 scored tasks across 10 application domains (3 skipped due to eval errors)
Max Steps: 15 per task
| Metric | Value |
|---|---|
| Success rate | 0/36 = 0% |
| CUA tasks | 30/36 (83%) |
| RPA false positives | 6/36 (17%) |
| Stuck-clicking rate | ~22/30 CUA tasks (73%) |
| Avg tokens | 35,916 |
| Avg time | 46.0s per task |
| Domain | Tasks | Mode | Stuck |
|---|---|---|---|
| chrome | 4 | all CUA | 4/4 |
| gimp | 2 | all CUA | 2/2 |
| libreoffice_calc | 3 | all CUA | 1/3 |
| libreoffice_impress | 2 | 1 CUA + 1 FP | 1/1 |
| libreoffice_writer | 2 | all CUA | 2/2 |
| multi_apps | 14 | 9 CUA + 5 FP | ~7/9 |
| os | 2 | all CUA | 2/2 |
| thunderbird | 2 | all CUA | 2/2 |
| vlc | 2 | all CUA | 2/2 |
| vs_code | 3 | all CUA | 2/3 |
Dominant failure: CUA stuck-clicking (67% of CUA tasks)
The LLM repeatedly sends identical pyautogui.click(x, y) coordinates. The auto-recovery mechanism (scroll, escape, Alt+Left after 4 identical actions) triggers but doesn't change screen state enough to break the loop.
RAG false positives (15% of all tasks)
6 tasks matched program 862006d6 ("Enter a list of names/items into a text editor and save the file") via word-overlap text matching. The program executes its 2-3 states, reaches terminal state, reports cov=100% but scores 0.0 because the program is completely irrelevant to the actual task.
SOTA comparison:
- UI-TARS-2: 47.5% on OSWorld-Verified
- OpenCUA-72B: 45.0% on OSWorld-Verified
- PreAct (Claude Sonnet CUA): 0%
- Desktop GUI screenshots are harder to parse than web/mobile — more visual complexity, smaller targets, deeper menus
- Accessibility tree truncation (3000 chars) misses critical elements
- No structured DOM equivalent — CUA relies entirely on visual + limited a11y
- Specialized vision models (UI-TARS, OpenCUA) trained specifically for desktop UI interaction outperform generic LLM CUA by a wide margin
Web (WebArena) > Mobile (AndroidWorld) > Desktop (OSWorld)
42% CUA 33-43% CUA 0% CUA
- Web: Structured DOM, clean layouts, standardized UI patterns, XPath verification works
- Mobile: Simpler visual layouts than desktop, but touch interactions (swipe, long-press) are harder than clicks; UI element data helps significantly
- Desktop: Most complex visual layouts, smallest click targets, deepest menu hierarchies, no structured DOM, complex multi-window interactions
| Component | WebArena | AndroidWorld | OSWorld |
|---|---|---|---|
| CUA agent | Works (42%) | Works for simple (43%) | Broken (0%) |
| Trajectory recording | Works | Works | Records failures |
| LLM compilation | Works | Works | Compiles failures |
| RAG storage | Works | Works | Works (but accumulates bad programs) |
| RAG retrieval | Works | False positives | False positives |
| RPA execution | Works (58% exec) | Not tested | N/A |
| Answer extraction | Bottleneck | N/A | N/A |
-
PreAct amplifies CUA quality: When CUA works, PreAct provides reliable 5x token savings. When CUA fails, PreAct has nothing to work with.
-
RAG text matching needs improvement: The word-overlap algorithm with 0.4 threshold produces false positives on all platforms. Stop-word filtering and higher thresholds are needed.
-
Platform-specific CUA models matter: Generic Claude Sonnet CUA works for web (42%) and simple mobile (43%) but fails completely on desktop (0%). SOTA desktop agents use specialized vision models.
-
Compilation is not the bottleneck: The LLM compiler reliably converts successful trajectories into executable state machines. The problem is generating successful trajectories in the first place.
-
Answer extraction is the web bottleneck: On WebArena, programs navigate correctly (58% exec) but extract wrong answers (29% eval). This is a compiler prompt issue, not a fundamental limitation.
- Fix RAG text matching: add stop-word filtering, raise threshold to 0.6+
- Fix answer extraction prompts for WebArena compiled programs
- Increase max steps for complex tasks (15 is too low for multi-app desktop tasks)
- Add "answer" action type recognition to AndroidWorld agent
- Fix separate RAG DB issue (relative persist_dir path)
- Add semantic validation to RPA terminal state (don't just return success=True)
- Implement domain-specific RAG matching (check application context before accepting matches)
- Use embedding-only RAG matching (remove text-overlap fast path)
- Expand a11y tree size limit for desktop environments
- Integrate specialized vision models (UI-TARS, OpenCUA) as CUA backend instead of generic LLM
- Build desktop-specific action primitives (menu navigation, dialog handling, multi-window management)
- Add screen region analysis to break stuck-clicking loops
- Consider hybrid approach: specialized vision for desktop screenshot parsing + LLM for task planning