diff --git a/docs/architecture/rfcs/loopx-overall-roadmap-v0.md b/docs/architecture/rfcs/loopx-overall-roadmap-v0.md index b611d832f0..5eb8b7d28e 100644 --- a/docs/architecture/rfcs/loopx-overall-roadmap-v0.md +++ b/docs/architecture/rfcs/loopx-overall-roadmap-v0.md @@ -219,6 +219,13 @@ onboarding time (including prerequisites/login) and the post-readiness interval; authentication or environment failures stay in the funnel. A small pilot is feedback, not statistical reliability or product-market fit. +The [independent App first-use/repeat-use protocol](../../product/use-cases/steward/first-use-evaluation.md) +prepares the five-slot freeze sheet, ordinary GQ01 handout, independent oracle +and separate onboarding/result/attention/cost observations. It reuses existing +optional first-run/usage-story intake and keeps failed or absent trials in the +denominator. This is preparation only: execution waits for a qualified pinned +package and consenting participants; no cohort result or recruitment is claimed. + The critical path is usable evidence presentation → a reproducible real episode → packaged first-use repair → independent reproduction. As of 2026-09-21, #4762 and #4811 are merged; #4811 records native Codex MCP execution and a real-model diff --git a/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md b/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md index 7331472718..e670b77bea 100644 --- a/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md +++ b/docs/architecture/rfcs/loopx-overall-roadmap-v0.zh-CN.md @@ -178,6 +178,12 @@ Muse 设计页在浏览器超时，其文章通过网页检索读取。本次调 性能数据。同时记录包含环境准备/登录的完整引导耗时与就绪后耗时，认证或环境 失败仍计入漏斗。小规模试用用于发现问题，不能据此宣称统计可靠性或 PMF。 +[App 独立首次/重复使用协议](../../product/use-cases/steward/first-use-evaluation.md) +已准备五个固定样本的冻结表、简短 GQ01 参与者请求、独立答案依据，以及分别记录 +引导/成果/注意力/成本的观察表。复用已有可选 first-run/usage-story 反馈，失败或 +未参与样本保留在分母中。目前仅完成协议准备；执行仍等待验收过的固定发行包和 +自愿参与者，不宣称试用结果，也未发起招募。 + 已合并 [#4814](https://github.com/loopx-project/loopx/pull/4814) 推进可选 S1/S5 入口： 管家和 Goal 飞书卡片共享 canonical 团队计划决策、认证投递绑定与重试恢复。真实 双卡交互单独验收；分配不是执行、采用或完整投研旅程。可读成果与本地首次使用 diff --git a/docs/product/public-adoption-loop.md b/docs/product/public-adoption-loop.md index 3424786976..39ba47574b 100644 --- a/docs/product/public-adoption-loop.md +++ b/docs/product/public-adoption-loop.md @@ -145,6 +145,19 @@ files that preserve the same public/private boundary. Until that approval exists, this document is the canonical adoption-loop copy. +## Independent App trials + +The generic workflow templates above remain proposed copy. The repository +already provides optional [First run](../../.github/ISSUE_TEMPLATE/first_run.yml) +and [Usage story](../../.github/ISSUE_TEMPLATE/usage_story.yml) feedback forms. +For the App's first-use cohort, reuse those forms with the +[frozen first-use and repeat-use protocol](use-cases/steward/first-use-evaluation.md). +Keep detailed timing, failures and evidence in a private study note; public +submission and anonymous citation remain separate opt-ins, never success gates. +The protocol is prepared, with no participant results. It requires a qualified +pinned package and consenting users before execution; it sends no invitations +and adds no telemetry or parallel issue intake. + ## Related Docs - [Release readiness](release-readiness.md) diff --git a/docs/product/use-cases/steward/first-use-evaluation.md b/docs/product/use-cases/steward/first-use-evaluation.md new file mode 100644 index 0000000000..383a441417 --- /dev/null +++ b/docs/product/use-cases/steward/first-use-evaluation.md @@ -0,0 +1,184 @@ +# Independent App first-use and repeat-use evaluation + +Status: **prepared protocol; no participant trials or passing results**. This +turns the [roadmap's alpha/beta targets](../../../architecture/rfcs/loopx-overall-roadmap-v0.md#delivery-order-owners-and-release-claims) +into a bounded study using the [golden-query pack](golden-queries.md). It adds no +runner, runtime obligation, telemetry or recruitment automation. Source tests +and scripted browser fixtures do not qualify an installed product. + +The question is whether a new person can obtain and find a useful result through +the App, then choose to return, without a maintainer operating their machine. +Start with one supported host/profile and one public task. Small-team, Lark, +cross-host and other-model claims retain their separate acceptance. + +## Freeze before the first attempt + +The evaluator copies this sheet into a private study note and fills every +applicable blank **before** a participant starts. The note remains private until +its owner approves a redacted summary. A document with blanks is preparation, +not a runnable or qualified cohort. + +| Frozen item | Required value | +| --- | --- | +| Study / package | Study revision, evaluation date/time zone, exact release artifact URL and digest, App/runtime revisions, previous known-good rollback package | +| Prerequisite qualification | Evidence that the selected installed package passed the supported entry/result/recovery path; an open implementation PR is insufficient | +| Supported configuration | OS/version, host, exact execution profile/model/effort, account type, documented prerequisites and supported install steps | +| Participants | Five consenting first-time users, anonymous slots U1–U5, prior relevant experience and declared independence from implementation; recruiting/contact details stay private | +| Task and sources | Fixed GQ01 request below, latest three completed fiscal years at the evaluation date, official report URLs/revisions or hashes, accessible public input copies | +| Independent oracle | Evaluator's checked fiscal periods, units, operating cash flow, cash purchases of property/equipment, definition of free cash flow and calculation; keep expected values away from the executing Agent | +| Envelope | Numeric model/tool call, token and time ceilings; monetary ceiling for metered usage, measurement source and unavailable-cost policy; no silent model substitution | +| Observation | Start/end definitions below, consented observation method, evaluator and scoring criteria, recovery/stop variant assigned before testing | +| Follow-up | Later calendar date/window for the same five users, repeat-task variant, consent and withdrawal handling; no reminder or invitation sent by this protocol | + +Do not start this cohort until the package, permitted usage and willing users +are available. A participant uses a fresh disposable project/Goal, not an active +personal or organizational Goal. Credentials stay in the normal machine-owned +login/configuration flow; the evaluator never copies them into a fixture. +Preconfiguration must match the shipped instructions. List every setup action; +do not hide maintainer-created bindings, repaired profiles or manual state edits. + +Freeze proposed targets of **10 minutes of configuration after prerequisites +and login**, **15 minutes from readiness to the first useful result**, **at least +4/5 successful first-use attempts**, and **at least 3/5 successful voluntary +repeat users on another day**. These are targets, not current measurements. Do +not adjust the task, thresholds or oracle after viewing candidate results. A +fix or package/profile change starts a separately identified cohort; preserve +the original five rows. + +For a subscription profile without task-level billing, freeze enforceable usage +and time ceilings and retain task cost as unknown. The artifact and usability +checks may pass; monetary-budget qualification remains unproven. Do not convert +an unknown subscription allocation into zero cost or a monetary-efficiency claim. + +## Participant handout / 给参与者的话 + +Use the selected release's ordinary install/login instructions. Open the LoopX +App and say: + +> 建个目标：研究微软近三年的现金流，给我一页报告。 +> +> Start a goal to study Microsoft's cash flow over the last three years. Give +> me a one-page report. + +The evaluator supplies the same pinned public reports and task budget through +the supported setup, not a hidden prompt that routes around the App. Equivalent +natural phrasing is welcome; the participant does not need Agent IDs, CLI +commands, internal receipt names or a prescribed answer. Confirm the actual +creation preview when it correctly represents the request. Ask an ordinary +clarification or change the request if needed; record it rather than coaching +the participant toward a successful score. + +> 结果留在这里，先别发布。卡住了就按界面提示处理；不想继续可以停止。 +> +> Keep the result here; don't publish it. Use the interface if something gets +> stuck, and stop whenever you want. + +Once the answer arrives, find its evidence, open the full result and reload the +conversation. The same result should still be readable. Say what you understood, +what was confusing and whether you would use LoopX again. Screen observation is +optional and requires consent; redact account details, paths and unrelated work. + +On the separately agreed later day, the participant may choose to return to the +same work and ask: + +> 接着上次的，看看资本开支增长对现金流的影响。 +> +> Pick up the previous report. Look at how growing capital expenditure affects +> cash flow. + +The repeat request should reuse the retained work and its constraints. It needs +checked calculations and limits, not a guaranteed investment forecast. Declining +or not returning is recorded with a volunteered reason, not replaced by another +user. This handout does not authorize public posting, trading, broader account +access or additional background work. + +## What counts as a useful result + +An evaluator who did not implement the candidate checks the stored result and +owning state after the participant finishes. Appearance or an Agent's claim of +completion does not decide the score. + +| Check | Evidence needed | +| --- | --- | +| Request survives | Original request, one confirmed Goal/execution path, no lost message, duplicate creation or competing driver; return stays in the initiating conversation | +| Work is understandable | Participant can identify actual owner and waiting/running/failed state, locate the result and source evidence, and explain any decision requested of them | +| Report is correct | All three frozen fiscal periods, consistent units, cash-flow/cash-capex table, explicit free-cash-flow definition, checked arithmetic, source links and uncertainty; leases/commitments are not silently counted as cash spending | +| Report is usable | Leading answer, a bounded comparison and readable Markdown or report; the participant can open it, find evidence and recover it after reload without a shell or manually relaying results | +| Envelope is respected | Observed usage fits the frozen enforceable ceilings and profile; report monetary coverage separately, and never use unknown cost to certify a monetary limit | +| Repeat actually works | A new later-day request uses the retained source/report version and no-publication constraint, produces a checked incremental result, and returns to the same conversation | + +Record **pass / fail / blocked / not run** per required check and surface. +Only pass counts as success. Record an unavailable prerequisite/profile as +blocked, a lost request or incorrect result as fail, and an unattempted task as +not run. Preserve abandonment and timeout at their observed stage. A result +rescued by maintainer shell work is not an independent first-use pass, even if +the repaired report is subsequently correct. + +Run GQ08 stop and GQ09 recovery companion variants on the same pinned package in +a disposable operator rehearsal before recruitment. Observe one interruption +and reconnection in the preassigned participant variant when consented. Stop +must affect the targeted work and report actual readback; do not infer team stop +from stopping the chat. Missing native evidence stays unqualified. These checks +do not replace G1's two real small-team cycles or authorize fault injection into +a participant's active work. + +## Observation sheet and honest denominators + +Use a monotonic elapsed clock during an attempt; record its date/time zone for +later-day repeat qualification. Keep these intervals separate: + +- **Complete onboarding:** first install/setup action to readiness, including + prerequisites and authentication. A pre-readiness failure still has a row. +- **Configuration:** prerequisites and login complete to readiness. Readiness + requires the actual supported profile/entry to be usable, not a green label. +- **First result:** submission after readiness to availability of the exact + result version subsequently accepted by the independent checker in the + original App conversation. Exclude retrospective scoring delay; a rejected + earlier version does not stop the clock. Report waiting and active attention too. +- **Recovery / repeat:** intervention to verified recovery, and later-day + request to its accepted incremental result. Unknown times remain unknown. + +For each intervention record stage, cause and actor. Separate required login, +explicit authorization, goal changes and voluntary deep discussion from avoidable +owner finding, context repetition, reminders and result transport. Count one +input once, retaining secondary tags. Normal documented UI actions are not +maintainer rescue; undocumented CLI/state repair or a maintainer taking over +the task is. An observer can ask for retrospective feedback after the attempt, +but hints during execution count as assistance. + +Copy this **unrun** ledger; do not treat its blank numbers as zero: + +| Slot | Onboarding / configuration / result time | Outcome / blocked stage | Assistance and attention | Measured cost or unknown | Evidence / later-day repeat | +| --- | --- | --- | --- | --- | --- | +| U1 | unknown | not run | unknown | unknown | none / not run | +| U2 | unknown | not run | unknown | unknown | none / not run | +| U3 | unknown | not run | unknown | unknown | none / not run | +| U4 | unknown | not run | unknown | unknown | none / not run | +| U5 | unknown | not run | unknown | unknown | none / not run | + +Publish only an approved, redacted summary: exact candidate/configuration, +attempted count out of the **frozen five**, accepted count out of five, each +failure/blocked/not-run row, later-day accepted repeat count out of those same +five, times, interventions and measured/unknown costs. Report timing coverage; +do not omit slow/failed samples or impute zero cost. For the 4/5 and 3/5 targets, +blocked, abandoned and not-run slots stay in the denominator. A useful-artifact +rate alone does not prove all safety, budget or advertised-platform gates passed. +Five users provide diagnostic feedback, not a statistical reliability claim. + +## Existing feedback route and next repair + +Keep detailed observations in the private study note. Participants may optionally +use the existing [First run](../../../../.github/ISSUE_TEMPLATE/first_run.yml) +or [Usage story](../../../../.github/ISSUE_TEMPLATE/usage_story.yml) forms for a +short public-safe outcome or failure description. Public issue submission and +anonymous citation are separate opt-ins; neither is required for participation +or a passing result. Do not paste Goal contents, raw transcripts, account data, +local paths or private evidence into the forms. + +Map each observed failure to the existing golden-query case and its owning +product task: creation/connection (GQ01/02), responsibility/return (GQ03/04), +continuity/controls (GQ07–09), readable result or evidence. Carry the reproduction, +expected result, package/profile and independent readback into that task rather +than creating a parallel adoption queue. Re-test on a new pinned candidate and +keep previous attempts visible. Preparing this protocol, filing feedback or +installing a package does not close the roadmap's product or cohort acceptance. diff --git a/docs/product/use-cases/steward/golden-queries.md b/docs/product/use-cases/steward/golden-queries.md index 7c7c9bde2d..14eaa23a60 100644 --- a/docs/product/use-cases/steward/golden-queries.md +++ b/docs/product/use-cases/steward/golden-queries.md @@ -391,3 +391,12 @@ Attach case-level evidence to the existing implementation PR and canonical Todo; only extract a durable regression into product tests after reproducing it. No speculative benchmark runner, public transcript corpus or additional polling automation is required to begin. + +After the installed journey is qualified, use the +[independent first-use and repeat-use protocol](first-use-evaluation.md) for the +roadmap's frozen five-person cohort. It supplies a short GQ01 participant request, +an independent report oracle, setup/result/attention/cost observations and an +unrun ledger. Preparation is allowed before a candidate is ready; participant +execution waits for the qualified pinned package and consent. Its 4/5 first-use +and 3/5 later-day repeat targets are not results and do not replace the P0 +routing, safety or small-team checks above.