diff --git a/docs/portfolio/validation/discovery-findings.md b/docs/portfolio/validation/discovery-findings.md new file mode 100644 index 0000000..0bf5b19 --- /dev/null +++ b/docs/portfolio/validation/discovery-findings.md @@ -0,0 +1,103 @@ +# Discovery Findings + +## Status + +**No external findings have been recorded.** All four hypotheses remain +**Unvalidated**. This file is a publication template, not evidence that research +has occurred. + +Update it only from pseudonymous private notes. Publish aggregates and safe +abstractions; never publish raw notes, names, employers, private links, internal +ticket names, incident details, screenshots, or credentials. + +## Sample composition + +| Metric | Verified count | +| --- | ---: | +| Eligible participants | 0 | +| Organizations represented | 0 | +| Release Managers | 0 | +| Technical Project Managers | 0 | +| Engineering Managers or Team Leads | 0 | +| Platform or DevOps leads | 0 | +| Participants outside current employer | 0 | +| Workflow observations | 0 | + +Recruitment targets are documented in [research-plan.md](research-plan.md). +Targets must not be copied into this table as achieved counts. + +## Evidence inventory + +| Evidence source | Verified | Self-reported | Notes | +| --- | ---: | ---: | --- | +| Discovery interviews | 0 | 0 | No interviews completed. | +| Workflow observations | 0 | 0 | No observations completed. | +| Concept tests | 0 | 0 | No concept tests completed. | + +## Recurring patterns + +Add a pattern only when at least two eligible participants support it. Report +the denominator and contradicting cases; do not infer market prevalence. + +| Pattern ID | Safe pattern statement | Supporting participants | Contradicting participants | Evidence mix | Implication | +| --- | --- | ---: | ---: | --- | --- | +| - | None recorded. | 0 | 0 | - | - | + +## Verified observations + +_None recorded._ + +For each future observation, state what was directly observed, the number of +participants or workflows, and the collection method. Do not merge observed +timing with recalled timing. + +## Self-reported evidence + +_None recorded._ + +For each future statement, show the participant count, sample denominator, role +mix when safe, and any uncertainty in recalled numbers. + +## Contradictions and negative evidence + +_None recorded._ + +Record evidence that weakens the target user definition, problem frequency, +workflow-fragmentation assumption, status model, or expected outcome. A +participant with an already-automated trusted workflow is evidence, not an +outlier to discard. + +## Unexpected findings + +_None recorded._ + +## Implications, not decisions + +_None recorded._ + +An implication describes what the evidence may mean. A product decision belongs +in [product-decisions.md](product-decisions.md) only after the change threshold +is met. + +## Hypothesis status + +| Hypothesis | Status | Supporting evidence | Weakening evidence | Next test | +| --- | --- | --- | --- | --- | +| H1 - Problem | Unvalidated | None | None | Problem-first interviews anchored in the last release. | +| H2 - Workflow fragmentation | Unvalidated | None | None | Last-release walkthroughs and workflow observations. | +| H3 - Solution usefulness | Unvalidated | None | None | Post-discovery concept tests. | +| H4 - Outcome | Unvalidated | None | None | Controlled manual-versus-ARI pilot. | + +Use `Unvalidated`, `Mixed`, `Weakening`, or `Gate met` for future status. `Gate +met` must link to the applicable pre-declared decision gate and counts. + +## Limitations + +- The planned sample is small, purposive, and vulnerable to selection bias. +- Self-reported time estimates are not observed measurements. +- Interviews reveal remembered workflows and interpretations, not necessarily + actual behavior. +- Synthetic or public reconstructions may differ from private production + workflows. +- Findings cannot be generalized beyond the observed sample without additional + evidence. diff --git a/docs/portfolio/validation/interview-guide.md b/docs/portfolio/validation/interview-guide.md new file mode 100644 index 0000000..1d83488 --- /dev/null +++ b/docs/portfolio/validation/interview-guide.md @@ -0,0 +1,183 @@ +# External Validation Interview Guide + +## Purpose and session shape + +Use this guide for 30-40 minute interviews with practitioners who participate +in planned production-release decisions. The sequence matters: complete problem +discovery and baseline questions before showing AI Release Intelligence (ARI). + +| Segment | Time | Product visible? | +| --- | ---: | --- | +| Consent and role fit | 3 minutes | No | +| Last actual release | 15 minutes | No | +| Quantitative baseline | 5 minutes | No | +| Concept test | 10 minutes | Yes | +| Wrap-up | 2-5 minutes | Yes | + +## Opening and consent + +1. Explain that this is research, not a sales call, and that critical feedback + is more useful than positive feedback. +2. Confirm that the participant has joined a planned release decision in the + previous six months. +3. Ask them not to share customer data, credentials, private URLs, internal + ticket names, or sensitive incident details. +4. Confirm whether note-taking, recording, and screen sharing are allowed. These + are separate permissions; default to notes only. +5. State that public reporting will be anonymized and aggregated. Do not promise + anonymity that cannot be maintained in the private research record. + +## Part A - Role and release context + +- What was your role in the most recent planned production release you joined? +- How often do you participate in planned releases? +- Who made the final Go/No-Go decision in that release? +- What could you decide yourself, and what required another person's approval? +- What counted as the release scope? + +If the participant answers with a general process, redirect: “Please anchor this +in the last release you can safely discuss. What happened first?” + +## Part A - Last actual release walkthrough + +Ask for events in sequence. Probe concrete actions before opinions. + +- What triggered the review, and when did you personally start? +- What did you open first? What did you do next? +- Which tools, pages, dashboards, documents, or messages did you use? +- How did you decide that the intended scope was complete? +- How did you connect scoped issues to pull requests and the candidate branch? +- Which CI or status checks did you inspect? How did you treat a failed check? +- How did you verify critical blockers and dependencies? +- Which before-, during-, or after-release actions had to be confirmed? +- Were migrations involved? What evidence made them ready or not ready? +- Were risks accepted? Who accepted them, and where was the decision recorded? +- Was a back-merge required? If so, how was it verified? +- Where did you compare evidence manually or carry information mentally? +- What was uncertain at the decision point? +- What went wrong, was missed, or nearly went wrong? +- What happened after the decision? + +Useful neutral probes: + +- “What did you look at to know that?” +- “Where was that recorded?” +- “What happened in this release, specifically?” +- “What would have changed the decision?” +- “Was that a required check or your personal practice?” + +## Part A - Quantitative baseline + +- About how long did the last review take end to end? +- Of that time, how much was active checking or reconciliation rather than + waiting for people or systems? +- What range would be more honest if one number is hard to recall? +- How many distinct surfaces or systems did you use? +- How often does a comparable review occur? +- Which checks were mandatory, policy-specific, or commonly skipped? +- How often did you revisit the same page or evidence? + +Label all recalled numbers **Self-reported**. Only clocked or reproducibly +recorded measurements are **Verified**. + +## Part B - Concept test + +Only now open the current public synthetic demo. Give the participant a short, +neutral task: “Use this report to decide what you would do next. Think aloud and +open any evidence you consider necessary.” Do not explain the intended answer +unless the participant is blocked by the demo mechanics rather than the concept. + +Observe before probing: + +- the first status, finding, and evidence link inspected; +- navigation back to GitHub or other manual checks; +- misunderstandings, hesitation, and unprompted trust concerns; +- whether the participant can state a decision and required actions. + +Then ask: + +- What would you distrust in this report? +- Which evidence did you want to open first, and why? +- What is missing for a real Go/No-Go decision in your workflow? +- Which rule is wrong or too rigid for the release you described? +- Which checks must be configurable, and who should control the policy? +- When would you fall back to a manual review? +- What false positive would make the product too noisy? +- What false negative would make it unacceptable? +- How does `NEEDS_DECISION` fit or conflict with your risk-acceptance process? +- How does `INSUFFICIENT_DATA` fit or conflict with your process? +- Would you use this as decision support? Choose `Yes`, `Yes, with manual + verification`, or `No`, and explain the boundary. + +Avoid “Do you like it?”, “Would automation be useful?”, feature pitching, and +questions that assume ARI solves the participant's problem. + +## Wrap-up + +- What is the most important thing the current product model gets wrong? +- Is there another release practitioner with direct decision experience whom I + should invite? +- May I contact you about an observed workflow or controlled pilot? +- May I use a short anonymized quotation? Record exact scope; silence is not + permission. + +Do not ask for willingness to pay unless pricing is a defined research question +for a later phase. + +## Private note-taking structure + +Use a pseudonymous participant file. Keep raw notes outside the public +repository. + +```text +Participant ID: +Interview date and researcher: +Eligibility basis: +Role and safe company context: +Release frequency: +Go/No-Go owner and participant authority: +Consent: notes / recording / screen sharing / quotation: + +LAST RELEASE +Sequence with timestamps where available: +Systems and surfaces: +Manual reconciliation steps: +Failed-CI treatment: +Blockers and dependencies: +Operations and migrations: +Accepted risks: +Back-merge: +Uncertainty, failure, or near-miss: + +BASELINE +Self-reported total elapsed time: +Self-reported active review time or range: +Self-reported surface count: +Review frequency: + +CONCEPT TEST +First evidence opened: +Observed behavior: +Trust concerns: +Missing critical input: +Policy/configuration need: +Manual fallback trigger: +Trust-destroying false positive: +Trust-destroying false negative: +Decision-support acceptance: + +EVIDENCE ENTRIES +- [Verified | Self-reported | Inference] Observation: + Source/timestamp: + Supports or contradicts: + Confidence/uncertainty: + +RESEARCHER SYNTHESIS +Contradicting evidence: +Unexpected finding: +Potential implication, not yet a decision: +Follow-up consent and action: +``` + +After the session, classify evidence before synthesizing it. One interview may +reveal a severe flaw, but it does not establish a recurring pattern by itself. diff --git a/docs/portfolio/validation/pilot-methodology.md b/docs/portfolio/validation/pilot-methodology.md new file mode 100644 index 0000000..1bcc77c --- /dev/null +++ b/docs/portfolio/validation/pilot-methodology.md @@ -0,0 +1,184 @@ +# Controlled Pilot Methodology + +## Status and objective + +**Pilot status:** Not started + +**Outcome hypothesis:** Unvalidated + +The pilot compares manual release-readiness review with ARI-assisted review on +small, comparable scenarios. Its purpose is to estimate active review effort, +critical-blocker recall, evidence coverage, and decision trust in this sample; +it is not a production-adoption or market-demand test. + +## Participants and unit of analysis + +Target 2-3 eligible release practitioners. The primary unit is one participant +completing one scenario under one condition. Report participants, scenarios, +and completed runs separately so repeated runs are not presented as independent +users. + +## Scenario design + +Create at least one matched pair of scenarios with equivalent complexity. Each +scenario must specify before the session: + +- repository and milestone scope; +- candidate branch and immutable commit SHA; +- required policy checks; +- known critical blockers and non-blocking findings; +- expected operations, migration, and back-merge evidence where applicable; +- expected final readiness status and required actions; +- direct evidence links or fixture identifiers; +- scenario version and reviewer of the ground truth. + +Prefer synthetic or public evidence. Do not import private production data into +the repository. Do not design every scenario around the current product's happy +path; include ambiguity, irrelevant failed checks, and incomplete data where +these reflect the research question. + +## Ground truth + +Two reviewers should independently inspect the scenario manifest and evidence, +then reconcile disagreements before the pilot. Freeze the agreed ground truth +by scenario version and commit SHA. The participant and facilitator must not +change it during a run. + +Ground truth contains: + +```text +expected decision +critical blocker IDs +non-blocking finding IDs +required evidence-check IDs +required actions +policy assumptions +``` + +If ground truth is discovered to be wrong or ambiguous, mark the run invalid, +record the reason, correct and version the scenario, and rerun only if feasible. +Do not score the participant against revised truth retroactively. + +## Conditions + +### Manual condition + +The participant uses the GitHub surfaces and scenario materials normally +available to a reviewer. They may take notes. ARI and its derived report are not +available. + +### ARI-assisted condition + +The participant starts from the ARI report and may open any linked or manual +evidence. Manual verification is allowed and must be observed; the test is +decision support, not blind reliance. + +## Assignment and order + +Use a counterbalanced crossover where sample size allows it: + +- Participant A: manual scenario 1, ARI-assisted scenario 2. +- Participant B: ARI-assisted scenario 1, manual scenario 2. +- Participant C: use the order that best balances completed runs. + +A participant must not review the same scenario in both conditions because +memory would contaminate time and recall. Record prior familiarity with ARI and +the scenario. Give only the minimum mechanics training needed to operate the +demo; do not teach the expected decision. + +## Timing rules + +Use a monotonic timer or timestamped recording when consent permits. + +- **Start:** the participant receives the scenario goal and can begin inspecting + evidence. +- **Stop:** the participant states a final Go/No-Go decision and required actions. +- **Active review:** time spent inspecting, navigating, reasoning, recording, or + asking scenario-clarification questions. +- **Waiting:** system loading, facilitator interruption, or other delay that does + not require participant review activity. Log separately. +- **Pause:** pause for technical failure unrelated to the condition; record the + duration and cause. + +Report active review time as the primary measure. Preserve total elapsed and +waiting time so exclusions are auditable. Do not stop the clock when the +participant manually verifies an ARI finding; that is part of assisted review. + +## Measures + +### Critical-blocker recall + +```text +correctly identified ground-truth critical blockers +--------------------------------------------------- +total ground-truth critical blockers +``` + +Report `not applicable` for a scenario with no critical blockers; do not treat a +zero denominator as 100%. + +### False blockers + +Count ground-truth non-blocking items that the participant incorrectly treats +as release blockers. Report counts, not only a rate. + +### Evidence coverage + +```text +required checks supported by evidence identified by the participant +-------------------------------------------------------------------- +total required ground-truth checks +``` + +### Decision correctness + +Exact match to the ground-truth final status plus identification of every +required critical action. Report status correctness and action completeness +separately when they differ. + +### Decision confidence and trust + +After each run collect: + +- confidence from 1 (not confident) to 5 (fully confident); +- decision-support acceptance: `Yes`, `Yes, with manual verification`, or `No`; +- evidence opened first and evidence still verified manually; +- what created or reduced trust; +- any finding or omission that would invalidate use in a real decision. + +Confidence and acceptance are **Self-reported**. Observed navigation and timing +are **Verified**. + +## Comparison method + +- Publish every valid run in a per-scenario table with participant IDs removed + or coarsened when re-identification risk exists. +- Compare condition medians for active review time, blocker recall, evidence + coverage, and confidence. +- For matched scenario pairs, also report the scenario-level and participant- + level deltas; do not combine unmatched scenarios without explanation. +- Calculate time reduction as `(manual median - assisted median) / manual median`. +- Report false blocker counts and every missed critical blocker explicitly. +- With 2-3 participants, use descriptive statistics only. Do not report + statistical significance or extrapolate to a population. + +## Decision rule and safety stop + +The pre-declared positive heuristic is a 30% or greater reduction in median +active review time, no reduction in critical-blocker recall, no silently missed +critical blocker, and participant willingness to use the report as decision +support. The 30% value is a **Target**, not an expected or current result. + +Any silently missed critical blocker is a safety-significant result. Stop using +the affected scenario for positive outcome claims, investigate whether ground +truth, product behavior, or facilitation caused it, and preserve the result. + +## Limitations to report + +- sample size and recruitment source; +- synthetic, public, or reconstructed scenario mix; +- scenario comparability and ground-truth review; +- participant familiarity with GitHub, ARI, and release policy; +- order, learning, and facilitator effects; +- technical failures and excluded time; +- difference between controlled decision support and production adoption. diff --git a/docs/portfolio/validation/pilot-results.md b/docs/portfolio/validation/pilot-results.md new file mode 100644 index 0000000..dca24a7 --- /dev/null +++ b/docs/portfolio/validation/pilot-results.md @@ -0,0 +1,76 @@ +# Controlled Pilot Results + +## Status + +**No pilot runs have been completed.** No time-saving, recall, trust, adoption, +or user-impact result is claimed. + +The preregistered measurement and comparison rules are in +[pilot-methodology.md](pilot-methodology.md). Replace placeholders only with +recorded data from valid runs; do not copy targets into result cells. + +## Sample and execution + +| Metric | Verified value | +| --- | ---: | +| Eligible participants | 0 | +| Scenario versions | 0 | +| Valid manual runs | 0 | +| Valid ARI-assisted runs | 0 | +| Invalid or excluded runs | 0 | + +## Per-run results + +| Run | Scenario version | Condition | Order | Active review time | Waiting time | Critical blockers found / total | False blockers | Evidence checks covered / total | Status correct | Required actions complete | Confidence (1-5) | Acceptance | +| --- | --- | --- | ---: | ---: | ---: | --- | ---: | --- | --- | --- | ---: | --- | +| - | - | - | - | - | - | - | - | - | - | - | - | No runs. | + +## Aggregate comparison + +| Metric | Manual | ARI-assisted | Delta | Interpretation boundary | +| --- | ---: | ---: | ---: | --- | +| Median active review time | - | - | - | Not measured. | +| Median total elapsed time | - | - | - | Not measured. | +| Critical-blocker recall | - | - | - | Not measured. | +| False blocker count | - | - | - | Not measured. | +| Evidence coverage | - | - | - | Not measured. | +| Correct final status | - | - | - | Not measured. | +| Complete required actions | - | - | - | Not measured. | +| Median confidence | - | - | - | Not measured. | + +If scenario pairs are not comparable, report them separately instead of +publishing a misleading aggregate. A scenario with no critical blockers has +`not applicable` recall and is excluded from the recall denominator. + +## Trust and decision-support observations + +_None recorded._ + +Future synthesis must distinguish observed behavior from participant statements +and report manual verification rather than treating it as failure by default. + +## Missed critical blockers and invalid runs + +_None recorded._ + +Every missed critical blocker, technical failure, ground-truth defect, and +excluded run must be listed with its effect on interpretation. + +## Decision-gate assessment + +| Gate component | Target | Observed | Status | +| --- | --- | --- | --- | +| Median active review time | Material reduction; 30%+ is the current heuristic | Not measured | Unvalidated | +| Critical-blocker recall | No reduction | Not measured | Unvalidated | +| Silently missed critical blockers | Zero | Not measured | Unvalidated | +| Decision-support acceptance | Willingness to use, with manual verification allowed | Not measured | Unvalidated | + +## Interpretation + +No interpretation is available before data collection. Future conclusions must +be limited to the observed sample and scenario versions. Controlled-pilot +behavior must not be described as production adoption. + +## Limitations + +_Complete after the pilot using the required list in the methodology._ diff --git a/docs/portfolio/validation/product-decisions.md b/docs/portfolio/validation/product-decisions.md new file mode 100644 index 0000000..dfc6c02 --- /dev/null +++ b/docs/portfolio/validation/product-decisions.md @@ -0,0 +1,79 @@ +# Research-Informed Product Decisions + +## Status and threshold + +**No external-evidence-driven product decision has been recorded.** + +Do not implement every request. A meaningful change requires either repeated +evidence from multiple eligible participants or one high-severity flaw that +invalidates a core workflow assumption. Integration mentions and preferences +do not cross this threshold by themselves. + +## Traceability register + +| Decision ID | Research evidence | Evidence class and sample | Contradictions | Implication | Product decision | GitHub issue | Implementation PR | Verification | Release | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| - | None recorded. | Unvalidated; n=0 | - | - | - | - | - | - | - | + +Public evidence should link to an aggregate section or pattern ID in +[discovery-findings.md](discovery-findings.md) or +[pilot-results.md](pilot-results.md), never to raw participant notes. + +## Decision record template + +### DNN - Decision title + +**Status:** Proposed | Accepted | Rejected | Deferred | Superseded + +**Date:** YYYY-MM-DD + +**Decision owner:** Product owner + +#### Research evidence + +- Public aggregate or pattern link: +- Evidence class: +- Supporting sample and roles: +- Repeated across: +- Contradicting or negative evidence: +- Why the evidence threshold is or is not met: + +#### Implication + +State which user, workflow step, domain assumption, trust boundary, or outcome +is affected. Keep the implication separate from the proposed solution. + +#### Options considered + +| Option | Expected value | Cost or risk | Evidence fit | +| --- | --- | --- | --- | +| Keep current behavior | - | - | - | +| Minimal change | - | - | - | +| Broader change | - | - | - | + +#### Decision and trade-off + +State what changes, what deliberately does not change, and why. Preserve +uncertainty and define a reversal trigger. + +#### Implementation trace + +- GitHub issue: +- Acceptance criteria: +- Pull request: +- Verification evidence: +- Release note: +- Released version: + +#### Post-change check + +- What evidence will show that the change addressed the finding? +- What would cause the decision to be reversed or revised? + +## Release rule + +Completing research does not justify `v0.2.0`. Create a new product release only +when external evidence drives a meaningful implemented and verified iteration. +Release notes must state the aggregate sample, what changed because of evidence, +what remains unvalidated, and any measured pilot result with its method and +limitations. diff --git a/docs/portfolio/validation/research-plan.md b/docs/portfolio/validation/research-plan.md new file mode 100644 index 0000000..94ac489 --- /dev/null +++ b/docs/portfolio/validation/research-plan.md @@ -0,0 +1,241 @@ +# External Validation Research Plan + +## Status and scope + +**Evidence status:** Unvalidated + +**Product version at research start:** `v0.1.0` + +**Research start:** 2026-09-03 + +This study tests whether the release-readiness problem and the current AI +Release Intelligence (ARI) workflow generalize beyond the author's own +release-management experience. It is designed to find disconfirming evidence +as well as supporting evidence. + +The repository's current public claims remain unchanged. The +[README](../../../README.md) and [product case](../product-case.md) already +state that the public demo and automated checks are synthetic or technical +evidence, while external adoption and measured human time savings remain +unvalidated. + +## Evidence classes + +Every note, synthesis, metric, and public claim must use one of these classes: + +| Class | Meaning | +| --- | --- | +| **Verified** | Directly observed, recorded with consent, or reproducibly measured. | +| **Self-reported** | Stated by a participant but not independently observed. | +| **Target** | A desired future outcome or decision threshold. | +| **Unvalidated** | Not yet supported by external evidence. | + +An inference may be used in analysis only when it is explicitly labelled as an +inference and linked to the underlying verified or self-reported evidence. An +estimate must never be presented as a measurement. + +## Research questions and hypotheses + +### H1 - Problem + +Release Managers and Technical Project Managers regularly spend meaningful +active time manually reconciling release scope, pull requests, CI checks, +blockers, and operational evidence. + +- **Supporting evidence:** repeated last-release accounts or observations of + manual reconciliation, with material active effort and operational frequency. +- **Weakening or falsifying evidence:** most eligible teams have reliable + automated readiness checks; manual work is rare, immaterial, or limited to an + unusual process; participants do not recognize the proposed workflow. + +### H2 - Workflow fragmentation + +Relevant release-readiness evidence is distributed across multiple GitHub +surfaces or related engineering systems and requires manual reconciliation. + +- **Supporting evidence:** multiple sources are opened and cross-checked during + the last actual review; evidence is carried mentally or copied into a separate + record. +- **Weakening or falsifying evidence:** a single trusted surface already provides + sufficient evidence; navigation across surfaces does not require meaningful + interpretation or reconciliation. + +### H3 - Solution usefulness + +A deterministic, evidence-linked readiness report with explicit statuses +`READY`, `NOT_READY`, `NEEDS_DECISION`, and `INSUFFICIENT_DATA` can support real +Go/No-Go decision workflows. + +- **Supporting evidence:** multiple eligible practitioners can map the report to + their last-release workflow, identify evidence they would inspect, and regard + the report as usable decision support. +- **Weakening or falsifying evidence:** the domain model omits a common critical + decision input; statuses conflict with real authority or risk handling; users + must reconstruct the review manually despite the report. + +### H4 - Outcome + +ARI reduces active human review effort without reducing critical-blocker recall +or decision trust. + +- **Supporting evidence:** controlled comparisons show a material reduction in + median active review time with no loss of critical-blocker recall, no silently + missed critical blocker, and acceptable decision-support trust. +- **Weakening or falsifying evidence:** review time does not improve materially; + recall or evidence coverage falls; a critical blocker is missed; participants + reject the report as decision support. +- **Claim rule:** H4 remains **Unvalidated** until measured in the pilot. Discovery + opinions cannot establish this outcome. + +## Sample and eligibility + +### Discovery target + +- 6-10 practitioners, with an initial scheduling target of 8. +- At least 3 participants outside the author's current employer where feasible. +- Priority roles: Release Manager; Technical Project Manager; Engineering + Manager or Team Lead with release ownership; Platform or DevOps lead who + participates in Go/No-Go. + +### Inclusion criteria + +A participant must have participated in at least one planned production release +decision during the previous six months and be able to reconstruct a recent +real review without disclosing protected information. + +### Exclusion criteria + +Exclude people whose knowledge is limited to general preferences, deployment +execution without release-decision participation, or releases too distant to +reconstruct reliably. Record exclusions privately; do not replace an ineligible +participant merely to protect a target ratio. + +## Four-phase method + +### Phase A - Problem discovery + +Run a 30-40 minute interview anchored in the participant's last actual release. +Do not show ARI until the problem-discovery and baseline questions are complete. +Capture the decision owner, sequence, evidence surfaces, manual checks, +self-reported active review time, uncertainty, failed-CI handling, blockers, +operations, migrations, accepted risks, and back-merges where applicable. + +### Phase B - Concept test + +Show the current public synthetic demo only after discovery. Probe distrust, +missing evidence, first evidence links opened, policy configurability, fallback +to manual review, and trust-destroying false positives or false negatives. Do +not treat liking, feature requests, or hypothetical willingness as validation. + +### Phase C - Workflow observation + +Observe 2-3 release-review workflows where feasible. Measure active review time, +surfaces visited, repeated navigation, checks performed, information held +mentally versus recorded, uncertainty, missed checks, and reconciliation versus +waiting time. Use screen sharing only with consent. If NDA constraints prevent +direct observation, use a synthetic or public reconstruction and label it. + +### Phase D - Controlled pilot + +Run 2-3 comparable manual-versus-ARI-assisted scenarios according to +[pilot-methodology.md](pilot-methodology.md). Do not optimize for the current +five-minute target before measuring actual behavior. + +## Measures and operational definitions + +| Measure | Definition | Evidence class | +| --- | --- | --- | +| Self-reported active review time | Participant's estimate for the last actual review; record range if uncertain. | Self-reported | +| Observed active review time | Clocked participant activity from review start to stated decision and required actions, excluding separately logged waiting. | Verified | +| Evidence surfaces | Distinct pages, views, tools, or systems used to reach the decision. | Verified when observed; otherwise Self-reported | +| Manual reconciliation | A human comparison or interpretation of two or more evidence items not already resolved by a trusted decision surface. | Verified when observed; otherwise Self-reported | +| Critical-blocker recall | Correctly identified ground-truth critical blockers divided by all ground-truth critical blockers. | Verified | +| False blockers | Non-blocking ground-truth items incorrectly classified as release blockers. | Verified | +| Evidence coverage | Required ground-truth checks for which the participant identifies sufficient supporting evidence, divided by all required checks. | Verified | +| Decision confidence | Participant rating from 1 (not confident) to 5 (fully confident), collected after each scenario. | Self-reported | +| Decision-support acceptance | `Yes`, `Yes, with manual verification`, or `No` after using the report. | Self-reported | + +Waiting time, total elapsed release lead time, and active review time must be +reported separately. + +## Decision gates + +These are pre-declared decision heuristics, not facts or statistical proof. + +### Problem-validation gate + +A useful positive signal requires all of the following: + +- at least 5 of the first 8 eligible practitioners perform meaningful manual + reconciliation; +- at least 4 of 8 describe it as recurring material friction; +- the workflow occurs frequently enough to have operational consequence; +- no dominant contradicting pattern makes the target user definition invalid. + +### Solution-validation gate + +A useful positive signal requires multiple practitioners to see the +evidence-linked report as usable in their workflow and no fundamental mismatch +between ARI's domain model and common release-review practice. Counts, roles, +contradictions, and missing critical inputs must be reported. + +### Pilot gate + +A useful positive signal requires: + +- a material reduction in median active review time; 30% or more is the current + **Target**, not a promised result; +- no reduction in critical-blocker recall; +- no critical blocker silently missed; +- willingness to use the report as decision support, even if manual verification + remains. + +Failure to cross a gate is a research result. It must not be hidden by changing +the threshold after data collection. + +## Analysis and synthesis rules + +- Preserve one row per eligible participant in the private tracker. +- Assign pseudonymous IDs before notes are taken. +- Separate direct observation, participant statement, and researcher inference. +- Add a recurring theme to public findings only when supported by at least two + eligible participants; show the exact supporting count and contradictions. +- Do not infer prevalence from this purposive qualitative sample. +- Do not average away a missed critical blocker. +- Keep discovery, observation, and pilot evidence separate. +- Freeze a dated public synthesis at each meaningful evidence milestone. + +## Product-change rule + +Recommend a meaningful product change only when it is supported by repeated +evidence from multiple participants or by one high-severity flaw that invalidates +a core workflow assumption. Each accepted change must trace: + +`research evidence -> implication -> product decision -> issue -> PR -> verification -> release` + +Mentions of integrations, notifications, or broader scope are not sufficient by +themselves. Jira, GitLab, Bitbucket, multi-repository releases, notifications, +and deployment orchestration remain non-goals unless the evidence changes the +product boundary. + +## Privacy, consent, and NDA handling + +- Keep names, contact data, employer names, raw notes, recordings, internal + issue names, production incident details, private-system screenshots, + customer data, and credentials outside the public repository. +- Ask permission before recording or observing a screen. Record the consent + scope privately. +- Stop or redirect discussion when protected details appear. Replace them with + the minimum safe abstraction in notes. +- Publish only anonymized aggregates. Do not use a quotation publicly without + explicit quotation approval. +- Keep personal-project evidence separate from employer experience and never + imply employer endorsement or product usage. + +## Publication policy + +Do not update recruiter-facing claims after individual interviews. The README, +product case, release notes, profile, and resume may change only at a meaningful +evidence milestone and must include sample, method, observed values, and +limitations appropriate to the claim. Research completion alone does not justify +`v0.2.0`; only a meaningful evidence-driven product iteration does.