qa: a repeatable Studio task benchmark — "build a repair-ticket app with no code", timed, counting red errors — as the Studio UX baseline #22137

Activity

1. objectstack-fleet

   added

   enhancementNew feature or requestNew feature or request

   pm:queue

   priority:p3

   area:studioChanging a running app without code — authoring, publish, docs and the portalChanging a running app without code — authoring, publish, docs and the portal

   on Oct 8, 2026
2. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Grade (triage): domain:devx added; priority:p3 and pm:queue stand.

   • Lands in docs/qa/ (the scenario script, the scoring sheet and the baseline run record), which is the domain:devx lane's (docs/qa/**).
   • It is a maintainer-dispatched mode:cloud task (「这个可以单卡一个云卡片推进」), so the claiming seat sends it to a cloud session.
   • Its cross-references (every red banner and dead end tied to an existing card, or filed) come back to triage as new cards.

   Triage seat (objectstack-wide, seat post #6015) · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-08T02:57Z. ⛔ Not a claim, ⛔ not a dispatch.
3. objectstack-fleet

   added

   domain:devx

   pm:dispatched

   and removed

   pm:queue

   on Oct 8, 2026
4. objectstack-fleet

   assigned

   os-zhuang

   on Oct 8, 2026
5. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Claim: PM loop round 1 (session seat taking over an unclaimed card on the maintainer's word: 「这个谁在处理？没人处理你可以接手你可以派云卡片」)
   Session: session_01D76mrPJrSSdaKRxR2rvrMG
   Account: os-zhuang
   Branch: claude/issue-22137-studio-task-benchmark
   Worktree: objectstack-issue-22137 (inside the cloud session's own container)
   Domain: domain:devx
   File surface: docs/qa/studio-benchmark/** only (new directory: scenario script, scoring sheet, baseline record, real-user session protocol). ⛔ Not docs/qa/platform-checklist/** — #22140 is in the merge queue there. Stop on breach; explain in the report.
   Container & model: L, mode:cloud, model: opus — the default tier; dispatch-gates --tier: "no path-derived mandate", so the PM's per-card call (browser-driven benchmark authoring)
   Clause-②: no
   Responsibility: n/a — not a defect card
   Thread-read: 6051250769
   Serial constraints cleared: objectstack-ai/objectstack#22140 (docs/qa/platform-checklist/**, queued) — disjoint by the file surface above; in-flight domain:devx claims (22131, 22085, 22082, 22080, 22014) declare scripts/, .github/workflows/ and .changeset/ — none touches docs/qa/

   ---

   Generated by Claude Code
6. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Dispatched 2026-10-08T03:28Z — round 1, mode:cloud: session session_01VenZErfhzG51sJBHRM1pWJ ("⚡ devx #22137 Studio task benchmark"), branch claude/issue-22137-studio-task-benchmark per the claim above. Delivery is a draft PR plus one os-dev-report comment on this card; the dispatching seat collects both, subscribes to the PR and archives the session once the card is final.

   ---

   Generated by Claude Code
7. objectstack-fleet commented on Oct 8, 2026

   

   objectstack-fleetbot

   on Oct 8, 2026

   ContributorAuthor

   More actions

   os-dev-report

   {
     "issue": 22137,
     "status": "done",
     "branch": "claude/issue-22137-studio-task-benchmark",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/22180",
     "session": "session_01VenZErfhzG51sJBHRM1pWJ — this cloud run (dispatch claim 6051545275 names this branch; verified before the first write)",
     "premise_still_valid": true,
     "summary": "All four acceptance items are in docs/qa/studio-benchmark/ (draft PR 22180, Fixes #22137). (1) SCENARIO.md, SCORING.md and SCORESHEET.md hold the fixed script and the scoring sheet. (2) The baseline is baselines/2026-10-08-8fc50b76-a58626c8.md, measured on objectstack 8fc50b7647d30db2a0837877cf251c1163a3239e and objectui a58626c88dc85954bd0af24f16ebb69454c03eee (pin and console stamp equal). Headline: completed 8/8 · 1511 s · 16 red errors · 13 failed writes · 10 code-shaped entries · 0 dead ends. (3) Every red error and code-shaped entry carries a card reference: objectui#11786, objectui#11772, objectui#11788 and objectstack#22032, or one of the findings below. objectui#11790 was not reached: Studio's Create app dialog built the navigation without opening Interfaces. (4) USER-SESSIONS.md is the protocol and kit for 3–5 real-author sessions. Every step finished in the console. Two pieces of rework the script does not name came up at step 8: granting the end user access to the new object, and rewriting a validation rule. The condition builder had compiled that rule to CEL the runtime cannot evaluate, so no ticket could ever be set to Done. Docs only; skip-changeset applied; PR assignee os-zhuang.",
     "tests": "Builds (under os-verify-lock): pnpm build --concurrency=2, then pnpm objectui:build → VERDICT command-exit 0; console reports 'dist ready … from objectui@a58626c88dc8'. Boot: OS_PORT=41873 objectstack dev --ui --seed-admin -p 41873 -d file:(scratch)/data.db → health 200. Every step was verified through REST: draft reads with ?state=draft&package=com.acme.repair; publish-drafts → 200 (3 times); published reads of the object, flow and app. Forged create with status=done and no due date → 400 'could not be evaluated (runtime: no such overload: !null)' before the repair, and 400 'Set a due date before marking the ticket Done.' after it. A Done write with a due date → 400 '!string' (reproduced 3 times: two forged requests and the end user's UI PATCH). GET /api/v1/data/sys_inbox_message as the technician → 1 row, 'Ticket Leaking tap in room 204 is Done'. Local gates: node scripts/pm/dispatch-gates.mjs --commands derived 12 families, all run at 686dc621, each exit 0: check-ci-filter-parity, check-closing-keyword-parity (and --self-test), check-comment-mask-corpus, lint check:doc-formula-expressions, check:cross-package-test-inputs, check:doc-authoring, check:driver-memory-census, check:gitlink-declared, check:nul-bytes, check:refd-timer-probe, check:watch-hint-literal. dispatch-gates --ran → 12 derived, 12 run, 0 NOT-MEASURED. No package touched, so no build closure or package test is owed. The pre-push hook accepted both commits' trailer pair. CI: not awaited, per the dispatch contract.",
     "mcp_calls": "12 — mcp__github__issue_read ×3 (22137 get + get_comments; one objectui read refused: repo not in session scope), mcp__github__search_issues ×9 (objectstack dedupe). No GitHub write tool. Also claude-code-remote add_repo ×2 (objectui read: already public; push: refused by the auto-mode classifier).",
     "api_writes": "3 relay strokes, all through scripts/pm as objectstack-fleet[bot]: (1) fleet-write/dispatch.mjs pr_create → POST /repos/objectstack-ai/objectstack/pulls (draft forced; 5566 bytes sent and stored identically) = PR 22180; (2) label-write.mjs → labels_add skip-changeset + assign os-zhuang on 22180 (read-back matches; size/m from the labeler kept); (3) post-stamped.mjs → this comment, POST /repos/objectstack-ai/objectstack/issues/22137/comments. Not REST: git push ×3 (empty branch probe, WIP, baseline). Unauthenticated GETs to api.github.com for four objectui issues were refused by the session proxy (reads; no data).",
     "open_questions": [
       {
         "question": "Where should the baseline live: a committed file in docs/qa/studio-benchmark/baselines/ or a qa-run-style issue?",
         "options": ["A: a committed markdown file keyed to both SHAs (done)", "B: a qa-run issue per release"],
         "recommendation": "A, because the next release has to diff its run against the previous one field by field, next to the script it scores. The qa-run title grammar (qa-rollup.mjs) counts checklist verdicts, and the checklist runs/ folder is git-ignored. Reasoning recorded in the README and the PR body."
       },
       {
         "question": "The objectui dedupe search the dispatch asked for is NOT MEASURED. This session cannot read objectstack-ai/objectui issues: MCP refuses the repo as out of scope, the add_repo push attach was refused by the auto-mode classifier, and the fleet --read path was refused too. Who runs the objectui dedupe for the three objectui findings below before they are filed?",
         "options": ["A: the PM seat runs it with objectui access before filing", "B: attach objectui to a later session and rerun"],
         "recommendation": "A — the dedupe words are in each finding; the pinned source line is named, so a filer can also check whether a newer objectui main already changed it."
       },
       {
         "question": "Counting conventions this run had to decide. One red error per red element per author gesture. A free-text field with a browser suggestion list counts as code-shaped. End-user record writes in step 8 count as failed writes. Unscripted rework is timed as sub-steps 8a/8b where it was found. Keep these for the next run?",
         "options": ["A: keep, as written in SCORING.md", "B: the maintainer adjusts before the second run (and the baseline is re-scored from its own tables)"],
         "recommendation": "A — they are written into SCORING.md, so the next run applies the same ones; every count in the baseline lists its rows, so re-scoring under other rules is mechanical."
       },
       {
         "question": "objectui#11790 (Interfaces gaps) was not reached: Studio's Create app dialog builds the app and its navigation without opening Interfaces. Should the fixed script require building navigation in Interfaces?",
         "options": ["A: keep the script fixed — what a low-code author would do first is what it measures", "B: change step 6 to name Interfaces (starts a new baseline)"],
         "recommendation": "A — changing the script resets the comparison; cover Interfaces in the human sessions or a separate script."
       }
     ],
     "out_of_scope_findings": [
       "class: a · reach: Studio → object Validations → condition builder 'record.due_date' + 'is empty / false' compiles to `record.status == 'done' && !record.due_date`. Publish answers 200, then every Done write is refused: PATCH /api/v1/data/repair_ticket/ID → 400 VALIDATION_FAILED reason unevaluable, fault 'runtime: no such overload: !string' (with a date set) or '!null' (without one), shown to the end user as a red toast. · evidence: objectui a58626c88dc8 packages/app-shell/src/views/metadata-admin/inspectors/ConditionBuilder.tsx:484 emits `!SUBJECT` for falsy, and line 483 emits a bare SUBJECT for truthy; baseline red error 16 and failed write 13; reproduced 3 times. The builder also has no 'is null' operator, so the only repair is raw CEL. The publish and save door gave no verdict: sibling of objectstack-ai/objectstack#22032 (open), which names validation conditions. Producer half lands in objectui. · dedupe: condition builder is empty negation CEL · no such overload !string validation rule · ConditionBuilder falsy operator non-boolean field",
       "class: a · reach: Studio → Automations → flow designer. The edge '+' and the node's 'Add connected node' both insert a create_record node without offering a type. The node inspector's Node Type list has no notify (the palette has it), and no gesture connects two existing nodes. In the baseline this produced 12× PUT /api/v1/meta/flow/NAME?mode=draft → 422 INVALID_METADATA, plus the edge-left-behind chips of objectui#11772. · evidence: baseline red errors 4–14 and warnings; insertOnEdge defaults type to 'create_record' (objectui a58626c8 FlowCanvas.tsx, insertOnEdge signature). Recovery took select Start → palette → Notify, delete the orphaned End, re-add End from Notify. · dedupe: flow designer insert node default create_record · node type dropdown missing notify · connect existing flow nodes edge",
       "class: a · reach: Studio → Automations → Start node 'Record updated' on an object that is still an unpublished draft. The entry-condition builder offers only 'previous' as a subject, because GET /api/v1/meta/object/repair_ticket answers 404 for a draft-only object, so the author must switch to Expression and type CEL. · evidence: baseline code-shaped entry 8; network log shows the 404s on the published read while the draft exists. · dedupe: flow entry condition builder draft object fields · condition subject list only previous · flow designer reads published object not draft",
       "carrier: none (承接者:无) · noted, not filed — machine-name inputs where a picker or a derived value is expected. Package ID is not derived from the display name. Picklist option values are not derived from labels: label first, value empty → red 'empty options' banner. Lookup 'Related object' and trigger 'Object' are free text with a browser suggestion list. The condition value for a select field is free text, and typing the label 'Done' compiled to `record.status == 'Done'` (observed, not executed: the stored value is 'done', so the rule would never fire). Baseline code-shaped entries 1–7. · dedupe: picklist value derived from label · lookup related object free text datalist · condition builder select value picker",
       "carrier: none (承接者:无) · noted, not filed — a published object grants end users nothing, and neither Studio nor the publish result says so. The end user meets 'You don't have access' (baseline red error 15, class missing-step); the grant takes Studio → Access plus Setup → Users. Deny-by-default is correct, so this is guidance, not a defect. · dedupe: publish new object no access end user prompt permission set",
       "carrier: none (承接者:无) · noted, not filed — observations, not reproduced twice. (1) Renaming a form group saves the label, but the canvas header keeps 'New group'; keys are new_group / new_group_2. (2) The flow toolbar keeps 'Trigger: autolaunched' after the Start node becomes Record updated. (3) Condition subjects show machine paths, not labels. (4) The end-user console makes background reads that answer 403 (meta/_drafts, sys_activity, sys_comment), and the record panel tells the user they cannot view activity or comments on their own ticket."
     ]
   }

   ---

   Generated by Claude Code
8. objectstack-fleet

   closed this as completedin #22180on Oct 8, 2026
9. objectstack-fleet

   removed

   pm:dispatched

   on Oct 8, 2026
10. akarma-synetal

    added a commit that references this issue on Oct 9, 2026

    docs(qa): a repeatable Studio task benchmark — repair-ticket scenario…

    ...

    This issue will close once commit 1e5d322 is merged into the 'main' branch.

    1e5d322