diff --git a/external/ai-agents/subagent-driven-development/SKILL.md b/external/ai-agents/subagent-driven-development/SKILL.md index d8ca081..aac35b9 100644 --- a/external/ai-agents/subagent-driven-development/SKILL.md +++ b/external/ai-agents/subagent-driven-development/SKILL.md @@ -14,7 +14,21 @@ Execute plan by dispatching a fresh implementer subagent per task, a task review **Narration:** between tool calls, narrate at most one short line — the ledger and the tool results carry the record. -**Continuous execution:** Do not pause to check in with your human partner between tasks. Execute all tasks from the plan without stopping. The only reasons to stop are: BLOCKED status you cannot resolve, ambiguity that genuinely prevents progress, or all tasks complete. "Should I continue?" prompts and progress summaries waste their time — they asked you to execute the plan, so execute it. +**Continuous execution:** Do not pause to check in with your human partner between tasks. Execute all tasks from the plan without stopping. The only reasons to stop are the four named below, or all tasks complete. "Should I continue?" prompts and progress summaries waste their time — they asked you to execute the plan, so execute it. + +**Rulings, not stalls.** A running plan does not wait on a human. Conflicts, +ambiguities, plan defects, a cap you would have asked to exceed — decide +them. The spec is the binding authority, the plan is its argument, and your +judgment settles what neither answers. Record every decision in the ledger as +`Ruling: — — `, and keep +going. A wrong ruling costs rework your human partner can see and undo; a +session parked on a question costs their whole day and buys nothing. + +Four things stop you, and only these: an irreversible or destructive +operation; a security-sensitive action; a side effect outside this worktree +that norms say you ask about first (a merge, a push to a shared branch, a +publish); and a plan so broken that every path forward is a guess. For those, +stop and ask. ## When to Use @@ -51,50 +65,121 @@ digraph process { subgraph cluster_per_task { label="Per Task"; "Dispatch implementer subagent (./implementer-prompt.md)" [shape=box]; - "Implementer subagent asks questions?" [shape=diamond]; + "Implementer asks questions?" [shape=diamond]; "Answer questions, provide context" [shape=box]; - "Implementer subagent implements, tests, commits, self-reviews" [shape=box]; - "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)" [shape=box]; - "Task reviewer reports spec ✅ and quality approved?" [shape=diamond]; - "Dispatch fix subagent for Critical/Important findings" [shape=box]; - "Mark task complete in todo list and progress ledger" [shape=box]; + "Implementer implements, tests, commits, self-reviews" [shape=box]; + "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)" [shape=box]; + "Spec ✅ and quality approved?" [shape=diamond]; + "Finding conflicts with plan text?" [shape=diamond]; + "Rule on the conflict, ledger the ruling" [shape=box]; + "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [shape=box]; + "Dispatch scoped re-review (./re-review-prompt.md)" [shape=box]; + "All findings addressed?" [shape=diamond]; + "R = 5?" [shape=diamond]; + "Adjudicate each open finding" [shape=box]; + "Any load-bearing finding?" [shape=diamond]; + "Rule and continue; stop only if every path forward is a guess" [shape=box]; + "Park findings in ledger with rulings" [shape=box]; + "Append completion to ledger, mark todo complete" [shape=box]; } - "Read plan, note context and global constraints, create todos" [shape=box]; + "Setup: worktree, ledger check, read plan, pre-flight review" [shape=box]; "More tasks remain?" [shape=diamond]; - "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" [shape=box]; + "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals" [shape=box]; + "Final review clean: delete this plan's workspace" [shape=box]; "Use superpowers:finishing-a-development-branch" [shape=box style=filled fillcolor=lightgreen]; - "Read plan, note context and global constraints, create todos" -> "Dispatch implementer subagent (./implementer-prompt.md)"; - "Dispatch implementer subagent (./implementer-prompt.md)" -> "Implementer subagent asks questions?"; - "Implementer subagent asks questions?" -> "Answer questions, provide context" [label="yes"]; - "Answer questions, provide context" -> "Dispatch implementer subagent (./implementer-prompt.md)"; - "Implementer subagent asks questions?" -> "Implementer subagent implements, tests, commits, self-reviews" [label="no"]; - "Implementer subagent implements, tests, commits, self-reviews" -> "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)"; - "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)" -> "Task reviewer reports spec ✅ and quality approved?"; - "Task reviewer reports spec ✅ and quality approved?" -> "Dispatch fix subagent for Critical/Important findings" [label="no"]; - "Dispatch fix subagent for Critical/Important findings" -> "Write diff file, dispatch task reviewer subagent (./task-reviewer-prompt.md)" [label="re-review"]; - "Task reviewer reports spec ✅ and quality approved?" -> "Mark task complete in todo list and progress ledger" [label="yes"]; - "Mark task complete in todo list and progress ledger" -> "More tasks remain?"; + "Setup: worktree, ledger check, read plan, pre-flight review" -> "Dispatch implementer subagent (./implementer-prompt.md)"; + "Dispatch implementer subagent (./implementer-prompt.md)" -> "Implementer asks questions?"; + "Implementer asks questions?" -> "Answer questions, provide context" [label="yes"]; + "Answer questions, provide context" -> "Implementer implements, tests, commits, self-reviews"; + "Implementer asks questions?" -> "Implementer implements, tests, commits, self-reviews" [label="no"]; + "Implementer implements, tests, commits, self-reviews" -> "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)"; + "Generate review package, dispatch task reviewer (./task-reviewer-prompt.md)" -> "Spec ✅ and quality approved?"; + "Spec ✅ and quality approved?" -> "Append completion to ledger, mark todo complete" [label="yes"]; + "Spec ✅ and quality approved?" -> "Finding conflicts with plan text?" [label="no"]; + "Finding conflicts with plan text?" -> "Rule on the conflict, ledger the ruling" [label="yes"]; + "Rule on the conflict, ledger the ruling" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model"; + "Finding conflicts with plan text?" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [label="no"]; + "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" -> "Dispatch scoped re-review (./re-review-prompt.md)"; + "Dispatch scoped re-review (./re-review-prompt.md)" -> "All findings addressed?"; + "All findings addressed?" -> "Append completion to ledger, mark todo complete" [label="yes"]; + "All findings addressed?" -> "R = 5?" [label="no"]; + "R = 5?" -> "Fix round R of 5: R≤3 resume implementer; R≥4 fresh implementer, more capable model" [label="no - next round"]; + "R = 5?" -> "Adjudicate each open finding" [label="yes - breaker trips"]; + "Adjudicate each open finding" -> "Any load-bearing finding?"; + "Any load-bearing finding?" -> "Rule and continue; stop only if every path forward is a guess" [label="yes"]; + "Any load-bearing finding?" -> "Park findings in ledger with rulings" [label="no"]; + "Park findings in ledger with rulings" -> "Append completion to ledger, mark todo complete"; + "Append completion to ledger, mark todo complete" -> "More tasks remain?"; "More tasks remain?" -> "Dispatch implementer subagent (./implementer-prompt.md)" [label="yes"]; - "More tasks remain?" -> "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" [label="no"]; - "Dispatch final code reviewer subagent (../requesting-code-review/code-reviewer.md)" -> "Use superpowers:finishing-a-development-branch"; + "More tasks remain?" -> "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" [label="no"]; + "Dispatch final code reviewer (../requesting-code-review/code-reviewer.md)" -> "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals"; + "Final findings? ONE fix dispatch, one scoped re-review, adjudicate residuals" -> "Final review clean: delete this plan's workspace"; + "Final review clean: delete this plan's workspace" -> "Use superpowers:finishing-a-development-branch"; } ``` -## Pre-Flight Plan Review +## Setup -Before dispatching Task 1, scan the plan once for conflicts: +Ensure the work happens in an isolated workspace: use +superpowers:using-git-worktrees to create one or verify the existing one. +Never start implementation on a main/master branch without your human +partner's explicit consent. + +Conversation memory does not survive compaction. In real sessions, +controllers that lost their place have re-dispatched entire completed task +sequences — the single most expensive failure observed. Track progress in +a ledger file, not only in todos. + +- Each plan owns a workspace: at skill start, run this skill's + `scripts/sdd-workspace PLAN_FILE` — it prints the plan's git-ignored + directory (`/.superpowers/sdd//`), home to + every artifact for THIS plan: ledger, briefs, reports, review packages. + Another plan's directory is never yours to read or write. +- Check for this plan's ledger at `/progress.md`. If its first + line names your plan file, tasks with a `Task : complete` line are DONE + — do not re-dispatch them; resume at the first task without one. A task + whose last line is a fix round is mid-loop: resume the loop at the next + round. A ledger whose first line names a different plan file — or a stray + ledger at the old flat path `.superpowers/sdd/progress.md` — is another + plan's progress: leave it in place and start your own, fresh. +- Create the ledger with its identity as the first line: + `# SDD ledger — plan: `. +- The ledger is your recovery map: the commits it names exist in git even + when your context no longer remembers creating them. After compaction, + trust the ledger and `git log` over your own recollection. +- `git clean -fdx` will destroy the workspace (it's git-ignored scratch); if + that happens, recover from `git log`. + +Read the plan once, note its context and Global Constraints, and create a +todo per task. If the plan names a Spec, read that too: the spec is the +authority the plan argues from, and conflicts inside the plan resolve +against it. A plan with no reachable spec gets a ledger note saying so — +rulings made without one are provisional. + +Before dispatching Task 1, scan the plan once for conflicts, writing down +what you checked as you check it: - tasks that contradict each other or the plan's Global Constraints - anything the plan explicitly mandates that the review rubric treats as a defect (a test that asserts nothing, verbatim duplication of a logic block) -Present everything you find to your human partner as one batched question — -each finding beside the plan text that mandates it, asking which governs — -before execution begins, not one interrupt per discovery mid-plan. If the -scan is clean, proceed without comment. The review loop remains the net for -conflicts that only emerge from implementation. +The scan's output is a table, not a verdict. One row for every pair of tasks +that share a file or an interface: the two tasks, what one produces against +what the other consumes, and what you found. One row for every task: whether +its own text agrees with itself — the tests it specifies against the code it +specifies, the files it creates against the files it later touches. "The scan +is clean" without those rows is not a scan you ran. + +Write the table to the ledger. Rule on everything you find before execution +begins — each finding against the plan text that mandates it — and record +each ruling in the ledger. If the scan is clean, proceed without comment. +Rule on each conflict it surfaces — the spec is the binding authority, the +plan is its argument — record the ruling beside its row, and dispatch +Task 1. The review loop remains the net for conflicts that only emerge from +implementation. ## Model Selection @@ -110,7 +195,11 @@ capable available model, not the session default. **Review tasks**: choose the model with the same judgment, scaled to the diff's size, complexity, and risk. A small mechanical diff does not need the -most capable model; a subtle concurrency change does. +most capable model; a subtle concurrency change does. Scoped re-reviews of +small fix diffs take a cheap-to-mid tier. + +**Fix-loop escalation (rounds 4-5)**: use a model at least one tier above +the implementer that got stuck. **Always specify the model explicitly when dispatching a subagent.** An omitted model inherits your session's model — often the most capable and @@ -129,11 +218,76 @@ that implementer. Single-file mechanical fixes also take the cheapest tier. - Touches multiple files with integration concerns → standard model - Requires design judgment or broad codebase understanding → most capable model -## Handling Implementer Status +## The Task Loop + +**Batch small same-shape work.** When the plan lists several tasks that are +each a small, independent edit of the same kind — the same one-line fix, +constant change, or field addition repeated across files — do not dispatch +one subagent per task. Compose ONE dispatch brief listing every file and +its change, send the whole batch to a single subagent, and review its diff +as one unit. Reserve one-dispatch-per-task for work that needs its own +judgment, its own tests, or its own review surface. + +Everything you paste into a dispatch prompt — and everything a subagent +prints back — stays resident in your context for the rest of the session +and is re-read on every later turn. Hand artifacts over as files. + +**Waiting on dispatched subagents:** never poll a wait interface with +short timeouts, and never sit in one silent, open-ended wait either. +While you have local work — ledger updates, packaging the next review, +reading reports — keep working; child results arrive on their own. +When you are genuinely idle, wait in bounded stretches (five to ten +minutes, where your platform allows), and between stretches post one +line of status and reconcile your live children: list them, and chase +any that finished without reporting. A bounded stretch keeps nearly +all of a long wait's efficiency while guaranteeing a stuck or lost +child is noticed within minutes, not at the end of the session. + +### 1. Dispatch the implementer + +Record BASE (`git rev-parse HEAD`) before dispatching — the review package +and fix-round diffs need it. + +- **Task brief:** before dispatching an implementer, run this skill's + `scripts/task-brief PLAN_FILE N` — it extracts the task's full text to a + uniquely named file and prints the path. Compose the dispatch so the + brief stays the single source of + requirements. Your dispatch should contain: (1) one line on where this + task fits in the project; (2) the brief path, introduced as "read this + first — it is your requirements, with the exact values to use verbatim"; + (3) interfaces and decisions from earlier tasks that the brief cannot + know; (4) your resolution of any ambiguity you noticed in the brief; + (5) the report-file path and report contract. Exact values (numbers, + magic strings, signatures, test cases) appear only in the brief. Never + make a subagent read the whole plan file. +- **Report file:** name the implementer's report file after the brief + (brief `…/task-N-brief.md` → report `…/task-N-report.md`) and put it in + the dispatch prompt. The implementer writes the full report there and + returns only status, commits, a one-line test summary, and concerns. +- A dispatch prompt describes one task, not the session's history. Do not + paste accumulated prior-task summaries ("state after Tasks 1-3") into + later dispatches — a real session's dispatch hit 42k chars of which 99% + was pasted history. A fresh subagent needs its task, the interfaces it + touches, and the global constraints. Nothing else. +- The dispatch carries the no-subagents contract (it is in the + implementer template): the implementer never dispatches subagents — + not helpers, and never a reviewer. Review arrives from you, after the + report. In real sessions, every reviewer a worker spawned duplicated + the task review the controller dispatched anyway — a full extra + review seat per task. +- If an earlier task parked a finding in the area this task touches, carry + a pointer to that ledger entry in the dispatch. +- Record the implementer's agent identity from the dispatch result — + fix-loop rounds 1-3 resume this agent. +- Never dispatch multiple implementation subagents in parallel (conflicts). + +Template: [implementer-prompt.md](implementer-prompt.md) + +### 2. Handle the report Implementer subagents report one of four statuses. Handle each appropriately: -**DONE:** Generate the review package (`scripts/review-package BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. +**DONE:** Generate the review package (`scripts/review-package PLAN_FILE BASE HEAD`, from this skill's directory — it prints the unique file path it wrote; BASE is the commit you recorded before dispatching the implementer — never `HEAD~1`, which silently drops all but the last commit of a multi-commit task), then dispatch the task reviewer with the printed path. **DONE_WITH_CONCERNS:** The implementer completed the work but flagged doubts. Read the concerns before proceeding. If the concerns are about correctness or scope, address them before review. If they're observations (e.g., "this file is getting large"), note them and proceed to review. @@ -143,24 +297,41 @@ Implementer subagents report one of four statuses. Handle each appropriately: 1. If it's a context problem, provide more context and re-dispatch with the same model 2. If the task requires more reasoning, re-dispatch with a more capable model 3. If the task is too large, break it into smaller pieces -4. If the plan itself is wrong, escalate to the human +4. If the plan itself is wrong, rule on the correction, ledger it, and re-dispatch with the ruling carried in the dispatch **Never** ignore an escalation or force the same model to retry without changes. If the implementer said it's stuck, something needs to change. -## Handling Reviewer ⚠️ Items +If the implementer asks questions — before starting or mid-task — answer +clearly and completely, provide additional context if needed, and don't +rush it into implementation. -The task reviewer may report "⚠️ Cannot verify from diff" items — requirements -that live in unchanged code or span tasks. These do not block the rest of the -review, but you must resolve each one yourself before marking the task -complete: you hold the plan and cross-task context the reviewer -lacks. If you confirm an item is a real gap, treat it as a failed spec -review — send it back to the implementer and re-review. - -## Constructing Reviewer Prompts +### 3. Review the task Per-task reviews are task-scoped gates. The broad review happens once, at the -final whole-branch review. When you fill a reviewer template: +final whole-branch review. Never skip the task review, and never accept a +report missing either verdict — spec compliance AND task quality are both +required. Implementer self-review never replaces the task review; both are +needed. +- Hand the reviewer its diff as a file: run this skill's + `scripts/review-package PLAN_FILE BASE HEAD` and pass the reviewer the file path + it prints (or, without bash: `git log --oneline`, `git diff --stat`, + and `git diff -U10` for the range, redirected to one uniquely named + file). The output never enters your own context, and the reviewer sees + the commit list, stat summary, and full diff with context in one Read + call. Use the BASE you recorded before dispatching the implementer — + never `HEAD~1`, which silently truncates multi-commit tasks. Never + dispatch a task reviewer without a diff file. +- **Reviewer inputs:** the task reviewer gets three paths — the same brief + file, the report file, and the review package — plus the global + constraints that bind the task. +- The global-constraints block you hand the reviewer is its attention + lens. Copy the binding requirements verbatim from the plan's Global + Constraints section or the spec: exact values, exact formats, and the + stated relationships between components ("same layout as X", "matches + Y"). The reviewer's template already carries the process rules (YAGNI, + test hygiene, review method) — the constraints block is for what THIS + project's spec demands. - Do not add open-ended directives like "check all uses" or "run race tests if useful" without a concrete, task-specific reason - Do not ask a reviewer to re-run tests the implementer already ran on the @@ -171,110 +342,172 @@ final whole-branch review. When you fill a reviewer template: loop. If the prompt you are writing contains "do not flag," "don't treat X as a defect," "at most Minor," or "the plan chose" — stop: you are pre-judging, usually to spare yourself a review loop. -- The global-constraints block you hand the reviewer is its attention - lens. Copy the binding requirements verbatim from the plan's Global - Constraints section or the spec: exact values, exact formats, and the - stated relationships between components ("same layout as X", "matches - Y"). The reviewer's template already carries the process rules (YAGNI, - test hygiene, review method) — the constraints block is for what THIS - project's spec demands. -- Hand the reviewer its diff as a file: run this skill's - `scripts/review-package BASE HEAD` and pass the reviewer the file path - it prints (or, without bash: `git log --oneline`, `git diff --stat`, - and `git diff -U10` for the range, redirected to one uniquely named - file). The output never enters your own context, and the reviewer sees - the commit list, stat summary, and full diff with context in one Read - call. Use the BASE you recorded before dispatching the implementer — - never `HEAD~1`, which silently truncates multi-commit tasks. -- A dispatch prompt describes one task, not the session's history. Do not - paste accumulated prior-task summaries ("state after Tasks 1-3") into - later dispatches — a real session's dispatch hit 42k chars of which 99% - was pasted history. A fresh subagent needs its task, the interfaces it - touches, and the global constraints. Nothing else. -- Dispatch fix subagents for Critical and Important findings. Record Minor - findings in the progress ledger as you go, and point the final - whole-branch review at that list so it can triage which must be fixed - before merge. A roll-up nobody reads is a silent discard. -- A finding labeled plan-mandated — or any finding that conflicts with - what the plan's text requires — is the human's decision, like any plan - contradiction: present the finding and the plan text, ask which governs. - Do not dismiss the finding because the plan mandates it, and do not - dispatch a fix that contradicts the plan without asking. -- The final whole-branch review gets a package too: run - `scripts/review-package MERGE_BASE HEAD` (MERGE_BASE = the commit the - branch started from, e.g. `git merge-base main HEAD`) and include the - printed path in the final review dispatch, so the final reviewer reads - one file instead of re-deriving the branch diff with git commands. -- Every fix dispatch carries the implementer contract: the fix subagent - re-runs the tests covering its change and reports the results. Name the - covering test files in the dispatch — a one-line fix does not need the - whole suite. Before re-dispatching the reviewer, confirm the fix report - contains the covering tests, the command run, and the output; dispatch - the re-review once all three are present. -- If the final whole-branch review returns findings, dispatch ONE fix - subagent with the complete findings list — not one fixer per finding. - Per-finding fixers each rebuild context and re-run suites; a real - session's final-review fix wave cost more than all its tasks combined. - -## File Handoffs - -Everything you paste into a dispatch prompt — and everything a subagent -prints back — stays resident in your context for the rest of the session -and is re-read on every later turn. Hand artifacts over as files: - -- **Task brief:** before dispatching an implementer, run this skill's - `scripts/task-brief PLAN_FILE N` — it extracts the task's full text to a - uniquely named file and prints the path. Compose the dispatch so the - brief stays the single source of requirements. Your dispatch should - contain: (1) one line on where this task fits in the project; (2) the - brief path, introduced as "read this first — it is your requirements, - with the exact values to use verbatim"; (3) interfaces and decisions - from earlier tasks that the brief cannot know; (4) your resolution of - any ambiguity you noticed in the brief; (5) the report-file path and - report contract. Exact values (numbers, magic strings, signatures, test - cases) appear only in the brief. -- **Report file:** name the implementer's report file after the brief - (brief `…/task-N-brief.md` → report `…/task-N-report.md`) and put it in - the dispatch prompt. The implementer writes the full report there and - returns only status, commits, a one-line test summary, and concerns. -- **Reviewer inputs:** the task reviewer gets three paths — the same brief - file, the report file, and the review package — plus the global - constraints that bind the task. -- Fix dispatches append their fix report (with test results) to the same - report file and return a short summary; re-reviews read the updated file. +The task reviewer may report "⚠️ Cannot verify from diff" items — requirements +that live in unchanged code or span tasks. These do not block the rest of the +review, but you must resolve each one yourself before marking the task +complete: you hold the plan and cross-task context the reviewer +lacks. If you confirm an item is a real gap, treat it as a failed spec +review — it enters the fix loop with the other findings. -## Durable Progress +Template: [task-reviewer-prompt.md](task-reviewer-prompt.md) -Conversation memory does not survive compaction. In real sessions, -controllers that lost their place have re-dispatched entire completed task -sequences — the single most expensive failure observed. Track progress in -a ledger file, not only in todos. +### 4. The fix loop -- At skill start, check for a ledger: - `cat "$(git rev-parse --show-toplevel)/.superpowers/sdd/progress.md"`. Tasks listed there - as complete are DONE — do not re-dispatch them; resume at the first task - not marked complete. -- When a task's review comes back clean, append one line to the ledger in - the same message as your other bookkeeping: - `Task N: complete (commits .., review clean)`. -- The ledger is your recovery map: the commits it names exist in git even - when your context no longer remembers creating them. After compaction, - trust the ledger and `git log` over your own recollection. -- `git clean -fdx` will destroy the ledger (it's git-ignored scratch); if - that happens, recover from `git log`. +The loop triggers when the review reports spec ❌, any Critical or Important +finding, or a ⚠️ item you confirmed as a real gap. -## Prompt Templates +Before the loop starts, two routes leave it immediately: -- [implementer-prompt.md](implementer-prompt.md) - Dispatch implementer subagent -- [task-reviewer-prompt.md](task-reviewer-prompt.md) - Dispatch task reviewer subagent (spec compliance + code quality) -- Final whole-branch review: use superpowers:requesting-code-review's [code-reviewer.md](../requesting-code-review/code-reviewer.md) +- Record Minor findings in the progress ledger as you go + (`Task : minor (deferred): `), and point the final + whole-branch review at that list so it can triage which must be fixed + before merge. A roll-up nobody reads is a silent discard. Minor findings + never enter the loop. +- A finding labeled plan-mandated — or any finding that conflicts with + what the plan's text requires — is yours to rule on: weigh the finding + against the plan text, decide with the spec as the binding authority, and + ledger the ruling before you act on it. Do not dismiss the finding because + the plan mandates it, and do not dispatch a fix that contradicts the plan + without a recorded ruling. +Everything else enters the loop. A fix round is one fix dispatch plus one +scoped re-review. Five rounds maximum per task: + +**Rounds 1-3 — resume the original implementer.** Send it the open findings +verbatim. Its context is intact: it knows the task, the code, and its own +choices. If your harness cannot send another message to a live subagent, +dispatch a fresh implementer carrying the brief path, the report-file path, +and the findings — the report file is the persistent memory either way. + +**Rounds 4-5 — dispatch a fresh implementer on a more capable model** (per +Model Selection), with the brief path, the report-file path, the open +findings, and this framing: "A prior implementer attempted this task +[N] times; you own it now. Read the report file for what was tried." A loop +that survives three resumes usually means the implementer cannot see its +own problem — fresh eyes and a capability bump in one move. + +**Every round, either way:** the implementer fixes, re-runs the tests +covering the amended code, appends its fix report to the same report file, +and returns the short contract. Before re-dispatching the reviewer, confirm +the fix report contains the covering tests, the command run, and the +output; dispatch the re-review once all three are present. Name the +covering test files in the fix message — a one-line fix does not need the +whole suite. + +**The re-review is scoped.** Run `scripts/review-package PLAN_FILE FIX_BASE HEAD` +where FIX_BASE is the head the previous review saw, and dispatch +[re-review-prompt.md](re-review-prompt.md) with the findings list, the +brief, the report file, and the printed diff path. The re-reviewer verdicts +each finding ADDRESSED or NOT ADDRESSED and flags new breakage in the fix +diff only. New Critical/Important breakage in the fix diff joins the open +findings list. Out-of-scope observations go to the ledger as deferred +minors — they never extend the loop. + +**After each round,** append to the ledger: +`Task : fix round /5 ( addressed, open — ; commits ..)` + +Never fix findings yourself in the controller session — your context stays +clean for coordination, and controller fixes skip review. + +**The breaker.** When round 5's re-review still leaves findings open, stop +dispatching. Adjudicate each open finding yourself — you hold the plan and +the cross-task context the reviewer lacks: + +- **The reviewer is wrong, or the point is contestable:** park it — + `Task : parked — — Ruling: `. The final + review sees both sides. +- **Real, but nothing downstream builds on it:** park it the same way, with + a ruling that says it's real and deferred. +- **Real and load-bearing** — a later task builds on it, or it reveals a + plan defect: rule on the smallest change that unblocks the dependent work, + ledger it as `Task : Ruling: — `, + and carry it into the next task's dispatch. Parking a structural failure + silently lets every dependent task build on it. Stop only when the defect + leaves every path forward a guess. + +Adjudicate only at the cap. Adjudicating earlier to end a loop is +pre-judging with a different name. Every adjudication is a ledger entry — +a silent discard is forbidden. + +### 5. Complete the task + +When the review comes back clean — or every open finding is parked with a +ruling at the cap — append the completion line to the ledger in the same +message as your other bookkeeping: + +- `Task : complete (commits .., review clean)` +- `Task : complete (commits .., parked)` after a + tripped breaker + +Then mark the todo complete and move on. Never move to the next task while +the review has open Critical/Important issues that are neither fixed nor +parked-with-ruling at the cap. + +## Final Review + +The final whole-branch review gets a package too: run +`scripts/review-package PLAN_FILE MERGE_BASE HEAD` (MERGE_BASE = the commit the +branch started from, e.g. `git merge-base main HEAD`) and include the +printed path in the final review dispatch, so the final reviewer reads +one file instead of re-deriving the branch diff with git commands. Dispatch +on the most capable available model (see Model Selection), using +superpowers:requesting-code-review's +[code-reviewer.md](../requesting-code-review/code-reviewer.md). Point it at +the ledger's deferred-minor and parked lines so it can triage which must be +fixed before merge. + +If the final whole-branch review returns findings, dispatch ONE fix subagent +with the complete findings list — not one fixer per finding. +Per-finding fixers each rebuild context and re-run suites; a real +session's final-review fix wave cost more than all its tasks combined. +Then run exactly one scoped re-review of the fix wave +(`scripts/review-package PLAN_FILE FIX_BASE HEAD` over the fix range, +[re-review-prompt.md](re-review-prompt.md)). +Adjudicate any residual findings as in the task loop's breaker: park with +rulings, or rule on the load-bearing ones and ledger what you decided. Only +the four classes above stop you here. There is no second fix wave — +residual load-bearing findings surface to your human partner when +finishing-a-development-branch presents the options. + +## Finish + +Before you delete anything, collect every ledger line containing `Ruling:` — +preflight rulings, parked findings, breaker adjudications, all of them — into +your final message under "Rulings I made", in the order you made them, each +with what it costs if wrong. The list is exhaustive: if the ledger holds a +ruling, the list holds it. That list is the only place the decisions you +took on your human partner's behalf reach them — they read it and rework +whatever you got wrong. A ruling that dies with the workspace was a decision +made in secret. + +When the final whole-branch review is clean and its fixes are merged, +delete this plan's workspace (`rm -rf `) — the git history is +the record now. Sibling directories belong to other plans; leave them +alone. + +Use superpowers:finishing-a-development-branch. + +## Common Rationalizations + +| Excuse | Reality | +|--------|---------| +| "Close enough on spec compliance" | Reviewer found spec gaps = not done. Fix or hit the cap and adjudicate — those are the only exits. | +| "I'll fix it myself, dispatching is overhead" | Controller fixes pollute your context and skip review. Resume the implementer. | +| "One more round will converge" | Past the cap, rounds don't converge — the failure is structural. Adjudicate and route. | +| "The reviewer will just find something new anyway" | Scoped re-reviews verify fixes; they cannot wander. New findings on untouched code go to the ledger, not the loop. | +| "This finding is obviously wrong, I'll drop it" | You adjudicate only at the cap, and every ruling is a ledger entry. Silent discards are forbidden. | +| "The fix was small, skip the re-review" | Unreviewed fixes are how regressions land. Every round ends with a scoped re-review. | +| "Reviews slow the loop down" | The loop without reviews is just unverified churn. Reviews are the loop's brakes and steering. | +| "Ledger bookkeeping is overhead" | The ledger is what survives compaction. Controllers without one have re-dispatched entire completed task sequences. | +| "The implementer spawned its own reviewer — free extra assurance" | It's a duplicate seat reviewing the same diff; the task review is the gate. A worker-spawned reviewer is a defect to flag, not rigor. | ## Example Workflow ``` You: I'm using Subagent-Driven Development to execute this plan. +[Setup: worktree verified] [Read plan file once: docs/superpowers/plans/feature-plan.md] +[Resolve workspace: scripts/sdd-workspace docs/superpowers/plans/feature-plan.md — no ledger inside, fresh start] [Create todos for all tasks] Task 1: Hook installation script @@ -285,134 +518,51 @@ Implementer: "Before I begin - should the hook be installed at user or system le You: "User level (~/.config/superpowers/hooks/)" -Implementer: "Got it. Implementing now..." -[Later] Implementer: +Implementer: [Later] - Implemented install-hook command - Added tests, 5/5 passing - Self-review: Found I missed --force flag, added it - Committed -[Run review-package, dispatch task reviewer with the printed path] +[Run review-package PLAN_FILE BASE HEAD; dispatch task reviewer with the printed path] Task reviewer: Spec ✅ - all requirements met, nothing extra. Strengths: Good test coverage, clean. Issues: None. Task quality: Approved. -[Mark Task 1 complete] +[Ledger: Task 1: complete (commits a1b2c3d..d4e5f6a, review clean)] Task 2: Recovery modes [Run task-brief for Task 2; dispatch implementer with brief + report paths + context] -Implementer: [No questions, proceeds] -Implementer: +Implementer: [No questions] - Added verify/repair modes - 8/8 tests passing - - Self-review: All good - Committed -[Run review-package, dispatch task reviewer with the printed path] +[Run review-package PLAN_FILE BASE HEAD; dispatch task reviewer with the printed path] Task reviewer: Spec ❌: - Missing: Progress reporting (spec says "report every 100 items") - - Extra: Added --json flag (not requested) Issues (Important): Magic number (100) -[Dispatch fix subagent with all findings] -Fixer: Removed --json flag, added progress reporting, extracted PROGRESS_INTERVAL constant +[Fix round 1: resume the implementer with both findings] +Implementer: Added progress reporting, extracted PROGRESS_INTERVAL constant. + Re-ran test/recovery.test.js — 10/10 passing. Fix report appended. -[Task reviewer reviews again] -Task reviewer: Spec ✅. Task quality: Approved. +[Run review-package PLAN_FILE FIX_BASE HEAD; dispatch scoped re-review] +Re-reviewer: Missing progress reporting — ADDRESSED (src/recovery.js:41). + Magic number — ADDRESSED (src/recovery.js:7). New breakage: none. + Verdict: all findings addressed. -[Mark Task 2 complete] +[Ledger: Task 2: fix round 1/5 (2 addressed, 0 open; commits d4e5f6a..b7c8d9e)] +[Ledger: Task 2: complete (commits d4e5f6a..b7c8d9e, review clean)] ... [After all tasks] -[Dispatch final code-reviewer] -Final reviewer: All requirements met, ready to merge +[Run review-package PLAN_FILE MERGE_BASE HEAD; dispatch final code-reviewer, most capable model] +Final reviewer: All requirements met. Deferred minors triaged: none block merge. -Done! -``` +[Delete this plan's workspace — the record now lives in git] -## Advantages - -**vs. Manual execution:** -- Subagents follow TDD naturally -- Fresh context per task (no confusion) -- Parallel-safe (subagents don't interfere) -- Subagent can ask questions (before AND during work) - -**vs. Executing Plans:** -- Same session (no handoff) -- Continuous progress (no waiting) -- Review checkpoints automatic - -**Efficiency gains:** -- Controller curates exactly what context is needed; bulk artifacts move - as files, not pasted text -- Subagent gets complete information upfront -- Questions surfaced before work begins (not after) - -**Quality gates:** -- Self-review catches issues before handoff -- Task review carries two verdicts: spec compliance and code quality -- Review loops ensure fixes actually work -- Spec compliance prevents over/under-building -- Code quality ensures implementation is well-built - -**Cost:** -- More subagent invocations (implementer + reviewer per task) -- Controller does more prep work (extracting all tasks upfront) -- Review loops add iterations -- But catches issues early (cheaper than debugging later) - -## Red Flags - -**Never:** -- Start implementation on main/master branch without explicit user consent -- Skip task review, or accept a report missing either verdict (spec compliance AND task quality are both required) -- Proceed with unfixed issues -- Dispatch multiple implementation subagents in parallel (conflicts) -- Make a subagent read the whole plan file (hand it its task brief — - `scripts/task-brief` — instead) -- Skip scene-setting context (subagent needs to understand where task fits) -- Ignore subagent questions (answer before letting them proceed) -- Accept "close enough" on spec compliance (reviewer found spec issues = not done) -- Skip review loops (reviewer found issues = implementer fixes = review again) -- Let implementer self-review replace actual review (both are needed) -- Tell a reviewer what not to flag, or pre-rate a finding's severity in the - dispatch prompt ("treat it as Minor at most") — the plan's example code is - a starting point, not evidence that its weaknesses were chosen -- Dispatch a task reviewer without a diff file — generate it first - (`scripts/review-package BASE HEAD`) and name the printed path in the - prompt -- Move to next task while the review has open Critical/Important issues -- Re-dispatch a task the progress ledger already marks complete — check - the ledger (and `git log`) after any compaction or resume - -**If subagent asks questions:** -- Answer clearly and completely -- Provide additional context if needed -- Don't rush them into implementation - -**If reviewer finds issues:** -- Implementer (same subagent) fixes them -- Reviewer reviews again -- Repeat until approved -- Don't skip the re-review - -**If subagent fails task:** -- Dispatch fix subagent with specific instructions -- Don't try to fix manually (context pollution) - -## Integration - -**Required workflow skills:** -- **superpowers:using-git-worktrees** - Ensures isolated workspace (creates one or verifies existing) -- **superpowers:writing-plans** - Creates the plan this skill executes -- **superpowers:requesting-code-review** - Code review template for the final whole-branch review -- **superpowers:finishing-a-development-branch** - Complete development after all tasks - -**Subagents should use:** -- **superpowers:test-driven-development** - Subagents follow TDD for each task - -**Alternative workflow:** -- **superpowers:executing-plans** - Use for parallel session instead of same-session execution +Done! Using superpowers:finishing-a-development-branch. +``` diff --git a/external/ai-agents/subagent-driven-development/implementer-prompt.md b/external/ai-agents/subagent-driven-development/implementer-prompt.md index 218fcfe..5c8ecd6 100644 --- a/external/ai-agents/subagent-driven-development/implementer-prompt.md +++ b/external/ai-agents/subagent-driven-development/implementer-prompt.md @@ -47,6 +47,18 @@ Subagent (general-purpose): While iterating, run the focused test for what you're changing; run the full suite once before committing, not after every edit. + ## You Do Not Dispatch Subagents + + Do all of this task's work yourself. Never spawn a subagent to + implement part of the task, and above all never spawn a reviewer to + check your work. Self-review (below) means reading your own diff. + Review is the controller's job: after you report, it dispatches a + fresh reviewer against your diff. A reviewer you spawn duplicates + that review at full cost, and its approval counts for nothing in + the process. If you catch yourself thinking "an independent review + would strengthen my report" — that review is already scheduled. + Report instead. + ## Code Organization You reason best about code you can hold in context at once, and your edits are more @@ -106,9 +118,12 @@ Subagent (general-purpose): ## After Review Findings - If a reviewer finds issues and you fix them, re-run the tests that cover - the amended code and append the results to your report file. Reviewers - will not re-run tests for you — your report is the test evidence. + If the task review finds issues, you will be resumed with the findings. + Fix them, re-run the tests that cover the amended code, and append a fix + report to your report file: what you changed, the covering tests you + ran, the command, and the output. Reviewers will not re-run tests for + you — your report is the test evidence. Then reply with the same short + status contract as your first report. ## Report Format diff --git a/external/ai-agents/subagent-driven-development/re-review-prompt.md b/external/ai-agents/subagent-driven-development/re-review-prompt.md new file mode 100644 index 0000000..ad74b10 --- /dev/null +++ b/external/ai-agents/subagent-driven-development/re-review-prompt.md @@ -0,0 +1,115 @@ +# Scoped Re-Review Prompt Template + +Use this template when dispatching a re-review after a fix round. The +re-reviewer verifies the findings were addressed and checks the fix diff for +new breakage. It is not a fresh review — the full review already happened. + +**Purpose:** Verify each finding from the previous review was addressed, and +that the fix itself broke nothing. + +``` +Subagent (general-purpose): + description: "Re-review Task N fix round R" + model: [MODEL — REQUIRED: choose per SKILL.md Model Selection; an omitted + model silently inherits the session's most expensive one] + prompt: | + You are re-reviewing one task's fix round. A previous review produced + findings; an implementer has attempted to fix them. Your job is to + verdict each finding and inspect the fix diff — nothing else. + + ## The Task + + Read the task brief: [BRIEF_FILE] + + ## The Findings Under Verification + + [FINDINGS] + + ## The Fix + + Read the implementer's report (fix reports are appended at the end): + [REPORT_FILE] + + **Fix base:** [FIX_BASE_SHA] (the head the previous review saw) + **Head:** [HEAD_SHA] + **Diff file:** [DIFF_FILE] + + Read the diff file once — it contains the fix commits, a stat summary, + and the fix diff with surrounding context. Do not re-run git commands. + If the diff file is missing, fetch the diff yourself: + `git diff --stat [FIX_BASE_SHA]..[HEAD_SHA]` and + `git diff [FIX_BASE_SHA]..[HEAD_SHA]`. + + Your review is read-only on this checkout. Do not mutate the working + tree, the index, HEAD, or branch state in any way. + + ## You Do Not Dispatch Subagents + + Do all of this review yourself. Never spawn a subagent to review part + of the diff, and never spawn another reviewer for a second opinion. + This process already provides every review seat the work gets; a + reviewer you spawn duplicates one of them at full cost, and its + verdict counts for nothing. If the diff feels too large for one + pass, review it in passes yourself and say so in your report. + + ## Scope + + Your scope is the findings list and the fix diff. Verdict every finding. + Inspect the fix diff for new problems the fix itself introduced. Do NOT + re-review code the fix did not touch: if you notice an issue entirely + outside the fix diff, report it under Out-of-Scope Observations — it + does not block this task and does not extend the loop. A broad + whole-branch review happens after all tasks are complete. + + ## Tests + + The implementer re-ran the tests covering the amended code and appended + the results to the report file. Treat the report as unverified claims: + confirm the fix report names the covering tests and shows their output, + and verify the claims against the diff. Do not re-run the suite to + confirm their report. Run a test only when reading the code raises a + specific doubt that no existing run answers — and then a focused test, + never a package-wide suite. + + ## Output Format + + Your final message is the report itself: begin directly with the first + finding's verdict. Every line is a verdict, a finding with file:line, + or a check you ran — no preamble, no process narration. + + ### Finding Verdicts + + For each finding in The Findings Under Verification, in order: + - **[finding one-liner]** — ADDRESSED | NOT ADDRESSED, with file:line + evidence. "Attempted" is not addressed: the specific defect must no + longer exist. + + ### New Breakage in the Fix Diff + + Anything the fix itself broke or introduced, with severity + (Critical/Important/Minor) and file:line. "None" if clean. + + ### Out-of-Scope Observations + + Issues you noticed entirely outside the fix diff. Non-blocking; the + controller ledgers these for the final review. "None" if none. + + ### Verdict + + **Fix round:** [All findings addressed, no new Critical/Important + breakage | Findings remain open] — list the open ones. +``` + +**Placeholders:** +- `[MODEL]` — REQUIRED: reviewer model per SKILL.md Model Selection; scoped + re-reviews of small fix diffs take a cheap-to-mid tier +- `[BRIEF_FILE]` — the task brief file (same file the implementer worked from) +- `[FINDINGS]` — the Critical/Important findings and spec gaps from the + previous review, copied verbatim, one per bullet +- `[REPORT_FILE]` — the implementer's report file (fix reports appended) +- `[FIX_BASE_SHA]` — the head the previous review saw +- `[HEAD_SHA]` — current commit +- `[DIFF_FILE]` — the path `scripts/review-package PLAN_FILE FIX_BASE HEAD` printed + +**Re-reviewer returns:** per-finding verdicts (ADDRESSED / NOT ADDRESSED), +new breakage in the fix diff, out-of-scope observations, and a round verdict. diff --git a/external/ai-agents/subagent-driven-development/scripts/review-package b/external/ai-agents/subagent-driven-development/scripts/review-package index 33bb20f..31852e2 100755 --- a/external/ai-agents/subagent-driven-development/scripts/review-package +++ b/external/ai-agents/subagent-driven-development/scripts/review-package @@ -4,26 +4,28 @@ # call. Using the recorded per-task BASE (not HEAD~1) keeps multi-commit # tasks intact. # -# Usage: review-package BASE HEAD [OUTFILE] -# Default OUTFILE: /.superpowers/sdd/review-...diff +# Usage: review-package PLAN_FILE BASE HEAD [OUTFILE] +# Default OUTFILE: /.superpowers/sdd//review-...diff # (named per range, so a re-review after fixes gets a distinct fresh file). set -euo pipefail -if [ $# -lt 2 ] || [ $# -gt 3 ]; then - echo "usage: review-package BASE HEAD [OUTFILE]" >&2 +if [ $# -lt 3 ] || [ $# -gt 4 ]; then + echo "usage: review-package PLAN_FILE BASE HEAD [OUTFILE]" >&2 exit 2 fi -base=$1 -head=$2 +plan=$1 +base=$2 +head=$3 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } git rev-parse --verify --quiet "$base" >/dev/null || { echo "bad BASE: $base" >&2; exit 2; } git rev-parse --verify --quiet "$head" >/dev/null || { echo "bad HEAD: $head" >&2; exit 2; } -if [ $# -eq 3 ]; then - out=$3 +if [ $# -eq 4 ]; then + out=$4 else - dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace") + dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan") out="$dir/review-$(git rev-parse --short "$base")..$(git rev-parse --short "$head").diff" fi diff --git a/external/ai-agents/subagent-driven-development/scripts/sdd-workspace b/external/ai-agents/subagent-driven-development/scripts/sdd-workspace index ea9bb08..4e2d168 100755 --- a/external/ai-agents/subagent-driven-development/scripts/sdd-workspace +++ b/external/ai-agents/subagent-driven-development/scripts/sdd-workspace @@ -1,22 +1,40 @@ #!/usr/bin/env bash -# Resolve and ensure the working-tree directory SDD uses for its short-lived -# artifacts: task briefs, implementer reports, review packages, and the -# progress ledger. Print the directory's absolute path. +# Resolve and ensure the working-tree directory SDD uses for one plan's +# short-lived artifacts: task briefs, implementer reports, review packages, +# and the progress ledger. Print the plan directory's absolute path. +# +# One directory per plan (.superpowers/sdd//) so a follow-up +# plan in the same working tree can never read or overwrite another plan's +# artifacts. A stale ledger misread as current progress makes controllers +# skip whole task sequences — plan-scoping removes that failure structurally. # # The workspace lives in the working tree (not under .git/) because Claude Code # treats .git/ as a protected path and denies agent writes there — which blocks # an implementer subagent from writing its report file. A self-ignoring -# .gitignore keeps the workspace out of `git status` and out of accidental -# commits without modifying any tracked file. +# .gitignore at .superpowers/sdd/ keeps every plan's workspace out of +# `git status` and out of accidental commits without modifying any tracked file. # # Single source of truth for the workspace location, so task-brief and # review-package cannot drift to different directories. # -# Usage: sdd-workspace +# Usage: sdd-workspace PLAN_FILE set -euo pipefail +if [ $# -ne 1 ]; then + echo "usage: sdd-workspace PLAN_FILE" >&2 + exit 2 +fi + +plan=$1 +[ -f "$plan" ] || { echo "no such plan file: $plan" >&2; exit 2; } + +slug=$(basename "$plan" .md) +[ -n "$slug" ] && [ "$slug" != "." ] && [ "$slug" != ".." ] \ + || { echo "cannot derive a workspace name from: $plan" >&2; exit 2; } + root=$(git rev-parse --show-toplevel) -dir="$root/.superpowers/sdd" +base="$root/.superpowers/sdd" +dir="$base/$slug" mkdir -p "$dir" -printf '*\n' > "$dir/.gitignore" +printf '*\n' > "$base/.gitignore" cd "$dir" && pwd diff --git a/external/ai-agents/subagent-driven-development/scripts/task-brief b/external/ai-agents/subagent-driven-development/scripts/task-brief index 247a767..612e14a 100755 --- a/external/ai-agents/subagent-driven-development/scripts/task-brief +++ b/external/ai-agents/subagent-driven-development/scripts/task-brief @@ -4,8 +4,9 @@ # through the controller's context. # # Usage: task-brief PLAN_FILE TASK_NUMBER [OUTFILE] -# Default OUTFILE: /.superpowers/sdd/task--brief.md -# (per worktree; concurrent runs in the same working tree share it). +# Default OUTFILE: /.superpowers/sdd//task--brief.md +# (per plan and per worktree; concurrent runs of the SAME plan in the same +# working tree share it). set -euo pipefail if [ $# -lt 2 ] || [ $# -gt 3 ]; then @@ -20,7 +21,7 @@ n=$2 if [ $# -eq 3 ]; then out=$3 else - dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace") + dir=$("$(cd "$(dirname "$0")" && pwd)/sdd-workspace" "$plan") out="$dir/task-${n}-brief.md" fi diff --git a/external/ai-agents/subagent-driven-development/task-reviewer-prompt.md b/external/ai-agents/subagent-driven-development/task-reviewer-prompt.md index 588a402..ce79694 100644 --- a/external/ai-agents/subagent-driven-development/task-reviewer-prompt.md +++ b/external/ai-agents/subagent-driven-development/task-reviewer-prompt.md @@ -52,6 +52,15 @@ Subagent (general-purpose): Your review is read-only on this checkout. Do not mutate the working tree, the index, HEAD, or branch state in any way. + ## You Do Not Dispatch Subagents + + Do all of this review yourself. Never spawn a subagent to review part + of the diff, and never spawn another reviewer for a second opinion. + This process already provides every review seat the work gets; a + reviewer you spawn duplicates one of them at full cost, and its + verdict counts for nothing. If the diff feels too large for one + pass, review it in passes yourself and say so in your report. + ## Do Not Trust the Report Treat the implementer's report as unverified claims about the code. It @@ -75,6 +84,13 @@ Subagent (general-purpose): Warnings or other noise in the implementer's reported test output are findings — test output should be pristine. + Evidence you cannot see is not evidence that doesn't exist. If the + report or its test evidence looks truncated, or you cannot locate the + results it claims, re-read the file at its stated path — and if it is + genuinely missing or garbled, report that as a gap for the controller. + Re-running the suite to regenerate what you failed to read is not + verification; illegibility of the evidence is not invalidation of it. + ## Part 1: Spec Compliance Compare the diff against What Was Requested: @@ -86,6 +102,12 @@ Subagent (general-purpose): - **Misunderstood:** right feature built the wrong way, wrong problem solved + If the brief lists several files each with its own change (a batched + dispatch), check the diff against that list file by file: every listed + file must have its corresponding hunk. A listed file the diff never + touches is a Missing finding, no matter how clean the rest of the + batch looks. + If a requirement cannot be verified from this diff alone (it lives in unchanged code or spans tasks), report it as a ⚠️ item instead of broadening your search. @@ -178,11 +200,8 @@ Subagent (general-purpose): - `[BASE_SHA]` — commit before this task - `[HEAD_SHA]` — current commit - `[DIFF_FILE]` — REQUIRED: the path the controller wrote the review - package to (`scripts/review-package BASE HEAD` prints the unique path it - wrote; the package never enters the controller's context) + package to (`scripts/review-package PLAN_FILE BASE HEAD` prints the unique + path it wrote; the package never enters the controller's context) **Reviewer returns:** Spec Compliance verdict (✅/❌/⚠️), Strengths, Issues (Critical/Important/Minor), Task quality verdict - -A fix dispatch can address spec gaps and quality findings together; -re-review after fixes covers both verdicts. diff --git a/external/basic/brainstorming/SKILL.md b/external/basic/brainstorming/SKILL.md index b0d52b2..b56a3b5 100644 --- a/external/basic/brainstorming/SKILL.md +++ b/external/basic/brainstorming/SKILL.md @@ -7,20 +7,91 @@ description: "You MUST use this before any creative work - creating features, bu Help turn ideas into fully formed designs and specs through natural collaborative dialogue. -Start by understanding the current project context, then ask questions one at a time to refine the idea. Once you understand what you're building, present the design and get user approval. +Start by classifying how much process the request needs, then work +through your path: understand the context, refine the idea, present a +design, and get your human partner's approval. -Do NOT invoke any implementation skill, write any code, scaffold any project, or take any implementation action until you have presented a design and the user has approved it. This applies to EVERY project regardless of perceived simplicity. +Do NOT invoke any implementation skill, write any code, scaffold any +project, or take any implementation action until you have told your +human partner what you intend and they have approved it. This applies +to EVERY task on EVERY path below — the ceremony scales with the task; +the approval gate never does. -## Anti-Pattern: "This Is Too Simple To Need A Design" - -Every project goes through this process. A todo list, a single-function utility, a config change — all of them. "Simple" projects are where unexamined assumptions cause the most wasted work. The design can be short (a few sentences for truly simple projects), but you MUST present it and get approval. +## Three Paths + +Before your first question, classify the request and say the +classification out loud — "this looks bounded, so I'll present a short +design here rather than write a spec" — so your human partner can +override it: + +- **Spike** — a feasibility question ("can we...", "is it possible...", + "quick and dirty is fine") whose output is an answer, not code you + keep. Present the question and what you'll try in 2-3 sentences, get + a nod, then find out as cheaply as correctness allows. No design + doc, no spec file. Report findings as a recommendation; anything you + built stays labeled throwaway. +- **Bounded** — a well-scoped change to code that already exists in + this repo: a new flag, a small endpoint, a one-file fix. + Understanding the kind of app is not enough — bounded means the flow + you are changing is already here to read. If there is no existing + flow to change, the task is not bounded. Ask the clarifying + questions that matter, present a short design IN CHAT (a few + sentences to a few short paragraphs), and STOP. Implementation + starts only after your human partner says yes to that design — a + bounded task's approval is as hard a gate as an architectural + one. No spec file, no implementation plan document. +- **Architectural** — new projects, new subsystems, changes that + restructure how components fit together or alter interfaces others + depend on. Follow the full process: questions, approaches, sectioned + design, written spec, then the writing-plans skill. + +When in doubt between two paths, take the heavier one. The ratchet is +one-way: hidden complexity discovered mid-task upgrades the path — +stop, say so, and step up. Nothing downgrades mid-task. + +## Anti-Pattern: "Too Simple To Need Approval" + +Every path ends with your human partner approving your intent before +implementation. A todo list, a single-function utility, a config +change — the design may be two sentences in chat, but you MUST present +it and get approval. "Simple" tasks are where unexamined assumptions +cause the most wasted work. What scales with simplicity is the +artifact, never the approval. + +## Red Flags + +| Thought | Reality | +|---------|---------| +| "This is too simple to need a design" | Simple means a short design, not no design. Two sentences in chat, then approval. | +| "I'll call it bounded and skip the spec" | Reaching for a label to skip work IS the doubt — take the heavier path. | +| "It's bounded and the design is obvious — I'll start while they read it" | The gate is the approval, not the design's length. Present, then stop until you hear yes. | +| "I understand this kind of app, so it's bounded" | Bounded measures the repo, not your familiarity. A new project has no existing flow — it is architectural. | +| "The spike works, so I'll keep the code" | A spike's output is an answer. Keeping the code is a new request — classify it. | +| "It grew, but I'm almost done — no need to re-classify" | Hidden complexity upgrades the path mid-task. Stop and say so. | +| "They approved the spike, so the follow-up change is approved too" | Each task gets its own classification and its own approval. | ## Checklist -You MUST create a task for each of these items and complete them in order: +Classify first, announce the path, then create a task for each item on +your path and complete them in order. + +**Spike:** +1. **Explore project context** — enough to frame the probe +2. **Present question + probe plan** — 2-3 sentences +3. **Get approval** — a nod is enough +4. **Investigate** — as cheaply as correctness allows +5. **Report findings** — a recommendation; label anything built as throwaway + +**Bounded:** +1. **Explore project context** — check files, docs, recent commits +2. **Ask clarifying questions** — one at a time, the ones that matter +3. **Present short design in chat** — approach, files touched, testing +4. **Get approval** — STOP and wait for an explicit yes; presenting the design and starting in the same breath is skipping the gate +5. **Implement** — proceed with the normal development workflow (TDD applies); no plan document +**Architectural:** 1. **Explore project context** — check files, docs, recent commits 2. **Offer the visual companion just-in-time** — NOT upfront. The first time a question would genuinely be clearer shown than described, offer it then (its own message); on approval its browser tab opens for you. If no visual question ever arises, never offer it. See the Visual Companion section below. 3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria @@ -35,6 +106,13 @@ You MUST create a task for each of these items and complete them in order: ```dot digraph brainstorming { + "Classify: spike / bounded / architectural" [shape=diamond]; + "Present question + probe (2-3 sentences)" [shape=box]; + "Ask clarifying questions (bounded)" [shape=box]; + "Present short design in chat" [shape=box]; + "Human approves?" [shape=diamond]; + "Investigate; report recommendation" [shape=doublecircle]; + "Implement via normal workflow (no plan doc)" [shape=doublecircle]; "Explore project context" [shape=box]; "Ask clarifying questions" [shape=box]; "Propose 2-3 approaches" [shape=box]; @@ -44,7 +122,17 @@ digraph brainstorming { "Spec self-review\n(fix inline)" [shape=box]; "User reviews spec?" [shape=diamond]; "Invoke writing-plans skill" [shape=doublecircle]; - + "Hidden complexity? Upgrade path" [shape=box]; + + "Classify: spike / bounded / architectural" -> "Present question + probe (2-3 sentences)" [label="spike"]; + "Classify: spike / bounded / architectural" -> "Ask clarifying questions (bounded)" [label="bounded"]; + "Classify: spike / bounded / architectural" -> "Explore project context" [label="architectural"]; + "Present question + probe (2-3 sentences)" -> "Human approves?"; + "Ask clarifying questions (bounded)" -> "Present short design in chat"; + "Present short design in chat" -> "Human approves?"; + "Human approves?" -> "Investigate; report recommendation" [label="spike: yes"]; + "Human approves?" -> "Implement via normal workflow (no plan doc)" [label="bounded: yes"]; + "Hidden complexity? Upgrade path" -> "Classify: spike / bounded / architectural"; "Explore project context" -> "Ask clarifying questions"; "Ask clarifying questions" -> "Propose 2-3 approaches"; "Propose 2-3 approaches" -> "Present design sections"; @@ -58,10 +146,21 @@ digraph brainstorming { } ``` -**The terminal state is invoking writing-plans.** Do NOT invoke frontend-design, mcp-builder, or any other implementation skill. The ONLY skill you invoke after brainstorming is writing-plans. +**Terminal states are path-bound.** Architectural: the ONLY skill you +invoke after brainstorming is writing-plans — never frontend-design, +mcp-builder, or any other implementation skill. Bounded: after +approval, implementation proceeds directly through the normal +development workflow; no plan document. Spike: the terminal state is a +reported recommendation. ## The Process +The subsections below serve the bounded and architectural paths (a +spike stops at "present the probe, get a nod"). Sections from +**Exploring approaches** onward are architectural-path depth — for +bounded work, context plus a few questions plus a short in-chat design +is the whole process. + **Understanding the idea:** - Check out the current project state first (files, docs, recent commits) @@ -77,6 +176,7 @@ digraph brainstorming { - Propose 2-3 different approaches with trade-offs - Present options conversationally with your recommendation and reasoning - Lead with your recommended option and explain why +- YAGNI ruthlessly - remove unnecessary features from every approach and design **Presenting the design:** @@ -99,7 +199,7 @@ digraph brainstorming { - Where existing code has problems that affect the work (e.g., a file that's grown too large, unclear boundaries, tangled responsibilities), include targeted improvements as part of the design - the way a good developer improves code they're working in. - Don't propose unrelated refactoring. Stay focused on what serves the current goal. -## After the Design +## After the Design (architectural path) **Documentation:** @@ -130,15 +230,6 @@ Wait for the user's response. If they request changes, make them and re-run the - Invoke the writing-plans skill to create a detailed implementation plan - Do NOT invoke any other skill. writing-plans is the next step. -## Key Principles - -- **One question at a time** - Don't overwhelm with multiple questions -- **Multiple choice preferred** - Easier to answer than open-ended when possible -- **YAGNI ruthlessly** - Remove unnecessary features from all designs -- **Explore alternatives** - Always propose 2-3 approaches before settling -- **Incremental validation** - Present design, get approval before moving on -- **Be flexible** - Go back and clarify when something doesn't make sense - ## Visual Companion A browser-based companion for showing mockups, diagrams, and visual options during brainstorming. Available as a tool — not a mode. Accepting the companion means it's available for questions that benefit from visual treatment; it does NOT mean every question goes through the browser. diff --git a/external/basic/brainstorming/visual-companion.md b/external/basic/brainstorming/visual-companion.md index 7b89f6b..c145e64 100644 --- a/external/basic/brainstorming/visual-companion.md +++ b/external/basic/brainstorming/visual-companion.md @@ -74,14 +74,22 @@ On Windows, the script auto-detects and switches to foreground mode (which block scripts/start-server.sh --project-dir /path/to/project --open ``` -**Copilot CLI:** +**Gemini CLI:** ```bash -# Use --foreground and start the server via the bash tool with mode: "async" -# so the process survives across turns. Capture the returned shellId for -# read_bash / stop_bash if you need to interact with it later. +# Use --foreground and set is_background: true on your shell tool call +# so the process survives across turns scripts/start-server.sh --project-dir /path/to/project --open --foreground ``` +**Copilot CLI:** +```bash +# Start it with Copilot CLI's non-blocking/background shell mechanism so the +# server survives across turns. Keep --foreground so the harness, not the +# script, owns backgrounding. The launcher is a .sh, so invoke it via bash +# (on Windows, call Git Bash's bash.exe from the PowerShell tool). +bash scripts/start-server.sh --project-dir /path/to/project --open --foreground +``` + **Other environments:** The server must keep running in the background across conversation turns. If your environment reaps detached processes, use `--foreground` and launch the command with your platform's background execution mechanism. If the URL is unreachable from your browser (common in remote/containerized setups), bind a non-loopback host: diff --git a/external/develop/code-quality/clean-code/SKILL.md b/external/develop/code-quality/clean-code/SKILL.md index ea6e121..9e7a61f 100644 --- a/external/develop/code-quality/clean-code/SKILL.md +++ b/external/develop/code-quality/clean-code/SKILL.md @@ -1,10 +1,15 @@ --- name: clean-code -description: Apply Robert C. Martin's Clean Code principles (naming, functions, comments, formatting, error handling, tests, classes, code smells). Use when writing new code, reviewing pull requests, refactoring legacy code, or aligning on team coding standards. +description: "This skill embodies the principles of \"Clean Code\" by Robert C. Martin (Uncle Bob). Use it to transform \"code that works\" into \"code that is clean.\"" +risk: safe +source: "ClawForge (https://github.com/jackjin1997/ClawForge)" +date_added: "2026-02-27" --- # Clean Code Skill +This skill embodies the principles of "Clean Code" by Robert C. Martin (Uncle Bob). Use it to transform "code that works" into "code that is clean." + ## 🧠 Core Philosophy > "Code is clean if it can be read, and enhanced by a developer other than its original author." — Grady Booch @@ -33,7 +38,7 @@ Use this skill when: ## 3. Comments - **Don't Comment Bad Code—Rewrite It**: Most comments are a sign of failure to express ourselves in code. -- **Explain Yourself in Code**: +- **Explain Yourself in Code**: ```python # Check if employee is eligible for full benefits if employee.flags & HOURLY and employee.age > 65: @@ -88,6 +93,12 @@ Use this skill when: - [ ] Am I passing too many arguments? - [ ] Is there a failing test for this change? +## Example + +**User request:** + +> Refactor this working code for clearer names, smaller units, explicit errors, and preserved behavior; verify it with focused tests. + ## Limitations - Use this skill only when the task clearly matches the scope described above. - Do not treat the output as a substitute for environment-specific validation, testing, or expert review. diff --git a/external/develop/devops/devops-engineer/SKILL.md b/external/develop/devops/devops-engineer/SKILL.md index 8b7004b..3629161 100644 --- a/external/develop/devops/devops-engineer/SKILL.md +++ b/external/develop/devops/devops-engineer/SKILL.md @@ -4,7 +4,7 @@ description: Creates Dockerfiles, configures CI/CD pipelines, writes Kubernetes license: MIT metadata: author: https://github.com/Jeffallan - version: "1.1.1" + version: "1.2.0" domain: devops triggers: DevOps, CI/CD, deployment, Docker, Kubernetes, Terraform, GitHub Actions, infrastructure, platform engineering, incident response, on-call, self-service role: engineer @@ -42,8 +42,9 @@ You are a senior DevOps engineer with 10+ years of experience. You operate with 2. **Design** - Pipeline structure, deployment strategy 3. **Implement** - IaC, Dockerfiles, CI/CD configs 4. **Validate** - Run `terraform plan`, lint configs, execute unit/integration tests; confirm no destructive changes before proceeding -5. **Deploy** - Roll out with verification; run smoke tests post-deployment -6. **Monitor** - Set up observability, alerts; confirm rollback procedure is ready before going live +5. **Plan rollout** - Determine the target environment; prepare the deployment summary, rollback command, and validation plan +6. **Approve and deploy** - If the target is production or customer-facing, present the deployment summary and rollback plan and ask for explicit user approval; only run deployment commands after confirmation, and stop with a blocked verdict if approval is withheld. Roll out with verification; run smoke tests post-deployment +7. **Monitor** - Set up observability, alerts; confirm rollback procedure is ready before going live ## Reference Guide @@ -52,6 +53,7 @@ Load detailed guidance based on context: | Topic | Reference | Load When | |-------|-----------|-----------| | GitHub Actions | `references/github-actions.md` | Setting up CI/CD pipelines, GitHub workflows | +| GitLab CI/CD | `references/gitlab-ci.md` | Setting up GitLab pipelines, `.gitlab-ci.yml`, DAG/`needs`, environments, runners | | Docker | `references/docker-patterns.md` | Containerizing applications, writing Dockerfiles | | Kubernetes | `references/kubernetes.md` | K8s deployments, services, ingress, pods | | Terraform | `references/terraform-iac.md` | Infrastructure as code, AWS/GCP provisioning | diff --git a/external/develop/devops/devops-engineer/references/gitlab-ci.md b/external/develop/devops/devops-engineer/references/gitlab-ci.md new file mode 100644 index 0000000..72c6e21 --- /dev/null +++ b/external/develop/devops/devops-engineer/references/gitlab-ci.md @@ -0,0 +1,192 @@ +# GitLab CI/CD Pipelines + +## Complete CI/CD Pipeline + +```yaml +# .gitlab-ci.yml — keep the root file short and declarative +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_TAG + - if: $CI_PIPELINE_SOURCE == "schedule" + +stages: [validate, test, build, deploy] + +default: + interruptible: true + retry: + max: 2 + when: [runner_system_failure, stuck_or_timeout_failure] + +include: + - local: .gitlab/ci/lint.yml + - local: .gitlab/ci/test.yml + - local: .gitlab/ci/build.yml + - local: .gitlab/ci/deploy.yml +``` + +```yaml +# .gitlab/ci/test.yml +unit-test: + stage: test + needs: [] # fails fast, doesn't wait on validate + image: node:20 + cache: + key: + files: [package-lock.json] + paths: [.npm/] + script: + - npm ci --cache .npm + - npm test -- --coverage + coverage: '/All files[^|]*\|[^|]*\s+([\d.]+)/' + artifacts: + reports: + junit: junit.xml + coverage_report: + coverage_format: cobertura + path: coverage/cobertura-coverage.xml +``` + +```yaml +# .gitlab/ci/build.yml +build-image: + stage: build + needs: [unit-test] + image: + name: moby/buildkit:rootless + entrypoint: [""] + variables: + BUILDKITD_FLAGS: --oci-worker-no-process-sandbox + before_script: + - mkdir -p ~/.docker + - AUTH=$(echo -n "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n') + - printf '{"auths":{"%s":{"auth":"%s"}}}' "$CI_REGISTRY" "$AUTH" > ~/.docker/config.json + script: + - > + buildctl-daemonless.sh build + --frontend dockerfile.v0 + --local context="${CI_PROJECT_DIR}" + --local dockerfile="${CI_PROJECT_DIR}" + --output type=image,name="${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}",push=true + --export-cache type=registry,ref="${CI_REGISTRY_IMAGE}:buildcache" + --import-cache type=registry,ref="${CI_REGISTRY_IMAGE}:buildcache" + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH +``` + +```yaml +# .gitlab/ci/deploy.yml +deploy-production: + stage: deploy + needs: [build-image] + environment: + name: production + url: https://app.example.com + deployment_tier: production + resource_group: production # serializes concurrent deploys + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: manual + script: + - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA +``` + +## Core Principles (from GitLab CI/CD best-practices survey) + +1. **`workflow:rules` is the master switch** — decide once, at the top, whether a pipeline should exist at all (push vs MR vs schedule vs API/trigger). Prevents duplicate pipelines from the same commit (e.g. branch pipeline + MR pipeline both firing) via `$CI_OPEN_MERGE_REQUESTS` / `$CI_MERGE_REQUEST_DRAFT` checks. +2. **`needs` over `stages` for DAG** — declare exact job dependencies with `needs:` instead of relying on stage ordering. Use `needs: []` for fast checks that should start immediately and fail early. Optimize the critical path, not every job. +3. **Cache vs artifacts are different things** — cache = reusable dependencies (key it off the lockfile, add `fallback_keys` so new branches don't start cold); artifacts = build outputs consumed by later jobs or humans (set `expire_in`, use `expose_as` for reviewer-facing files). Keep separate cache keys for protected vs unprotected refs. +4. **Reuse via `extends` + hidden jobs first, CI/CD components second** — components take typed `inputs` and should be pinned to a tag/SHA (never a moving `~latest`), documented, and treated as a supply-chain dependency if sourced externally. +5. **Environments are first-class objects** — declare `environment:name/url/deployment_tier`, use `on_stop`/`auto_stop_in` for ephemeral review apps, `resource_group` to serialize deploys to the same target, and protected environments + `manual_confirmation` for production. +6. **Secrets never live in CI/CD variables for anything sensitive** — prefer OIDC to cloud providers over static keys; use HashiCorp Vault integration with scoped roles, bound claims, and short TTLs. Be careful with `CI_JOB_TOKEN` scope (limit the allow-list) and treat MRs from forks as untrusted. +7. **Runner blast radius** — register runners at the narrowest scope that works (project < group < instance). Docker executor without `privileged` mode is the default; privileged/DinD only on isolated, ephemeral runners. Split protected and unprotected jobs onto separate runner pools/tags. +8. **Build images without long-lived root daemons** — prefer BuildKit rootless over classic privileged DinD (kaniko is archived and unmaintained; plan a migration if pipelines still use it); pass secrets via mount-type (`--mount=type=secret`), not `ARG`/`ENV`; tag by commit SHA, never `latest`; generate SBOM and sign images (cosign/notation) after build. +9. **MR pipelines are where checks matter** — prefer merged-results pipelines (test the merge of source+target, not just source) and merge trains for high-throughput repos, over relying on branch pipelines. +10. **Report everything GitLab can render** — `artifacts:reports:junit` for test results, `coverage_report` (Cobertura) for MR diff coverage annotations vs the `coverage:` regex for the summary badge, Code Quality reports, and screenshots/videos/logs as artifacts for UI/E2E failures. The MR widget is the primary feedback surface — optimize for it. + +## Common Patterns + +### Merge request pipelines only (no duplicate branch pipelines) +```yaml +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' +``` + +### Matrix builds +```yaml +test: + stage: test + parallel: + matrix: + - NODE_VERSION: ["18", "20", "22"] + OS: [ubuntu, alpine] + image: node:${NODE_VERSION}-${OS} +``` + +### Reusable CI/CD component (pinned, not ~latest) +```yaml +include: + - component: gitlab.com/my-org/ci-components/deploy@1.4.2 + inputs: + environment: production + k8s-namespace: app-prod +``` + +### Parent/child (monorepo) pipeline +```yaml +trigger-backend: + trigger: + include: backend/.gitlab-ci.yml + strategy: mirror # parent status mirrors child's real status + rules: + - changes: [backend/**/*] +``` + +### OIDC to a cloud provider (no static keys) +```yaml +deploy: + id_tokens: + AWS_ID_TOKEN: + aud: https://gitlab.example.com + script: + - aws sts assume-role-with-web-identity --role-arn $ROLE_ARN --web-identity-token $AWS_ID_TOKEN ... +``` + +### Dependency Proxy for base images (avoid Docker Hub rate limits) +```yaml +build: + image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:20-alpine +``` + +## Quick Reference + +| Feature | Purpose | +|---|---| +| `workflow:rules` | Decide whether a pipeline is created at all; dedupe push/MR pipelines | +| `needs:` | DAG dependencies between jobs, decoupled from stage order | +| `needs: []` | Job starts immediately, no upstream wait | +| `resource_group` | Serialize deploys to the same environment | +| `environment:deployment_tier` | Classifies env as production/staging/testing/development/other | +| `parallel:matrix` | Fan out a job across variable combinations | +| `extends` | Share config between jobs without `include` overhead | +| CI/CD components (`include:component`) | Typed, versioned, reusable pipeline building blocks | +| `trigger:include` + `strategy: mirror` | Parent/child pipelines for monorepos with real status propagation | +| `id_tokens` (OIDC) | Short-lived cloud credentials instead of static secrets | +| `artifacts:reports:junit` | Test results surfaced in MR widget | +| `artifacts:reports:coverage_report` | Per-line diff coverage annotations in MR | +| `CI_JOB_TOKEN` scope allow-list | Limits which projects a job token can access | + +## Anti-Patterns to Avoid + +- Hardcoding branch names/environments deep inside job scripts instead of centralizing in `workflow:rules` +- Letting `default:` become a dumping ground that obscures per-job behavior +- Relying on `stages:` ordering alone instead of `needs:` for large pipelines (slow, unclear critical path) +- One shared cache key for protected and unprotected refs (cache poisoning risk) +- Privileged Docker-in-Docker as the default build method +- Long-lived static cloud credentials in CI/CD variables when OIDC is available +- Including third-party CI/CD components at a floating ref instead of a pinned tag/SHA +- Testing only the source branch in MRs instead of merged results diff --git a/external/develop/devops/gcloud/SKILL.md b/external/develop/devops/gcloud/SKILL.md index 5f1f0c8..0d3a184 100644 --- a/external/develop/devops/gcloud/SKILL.md +++ b/external/develop/devops/gcloud/SKILL.md @@ -1,90 +1,107 @@ --- name: gcloud metadata: - category: DevOps + category: CloudInfrastructureAndServices description: >- - Interacts with Google Cloud services using the gcloud CLI safely and - efficiently. Covers command validation, data reduction, safety guardrails with - a denylist, and workflows for discovery and investigation. You MUST read this - skill before invoking any gcloud command. Use when managing cloud resources, - querying configurations, or troubleshooting issues via gcloud. Don't use when - writing or debugging Google Cloud client library code or raw REST/gRPC API - interactions. + Provides safety-critical validation, guardrails, and data reduction for gcloud + CLI operations across Google Cloud Platform (GCP) services and infrastructure. + Use when planning, generating, constructing, proposing, describing, or + executing any gcloud CLI commands - including when answering questions about + gcloud syntax, or formatting flags. Don't use when writing Google Cloud + client library code or raw REST/gRPC API requests. --- # gcloud CLI Skill for AI Agents +> [!CAUTION] +> +> ### MANDATORY PRE-CONDITION: EXPLICIT LEAF-LEVEL SYNTAX VALIDATION +> +> All pre-existing knowledge of `gcloud` commands, flags, flag values, and +> positional argument syntax is **stale and prone to hallucination**. +> +> NEVER propose command parameters, output flag options, execute commands, OR +> outline step-by-step plans for any `gcloud` task before validating leaf-level +> syntax via `gcloud help ` (or including leaf-level help lookup as a +> mandatory step in the plan). +> +> **Mandatory Action Rules**: +> +> 1. **Direct Execution & Code Generation**: **ALWAYS** invoke `gcloud help +> ` (e.g. `gcloud help compute instances create` or `gcloud +> help sql instances create`) before proposing or executing the final +> command syntax. +> +> 2. **Planning & Strategy Queries**: When asked for a plan, strategy, or next +> steps to achieve a user goal (e.g., *"What is your plan to accomplish +> X..."*), the response **MUST explicitly include running `gcloud help +> `** as Step 1 of the plan before proposing flags or +> executing commands. +> +> 3. **Non-Transitive Validation**: Parent command group help (e.g. `gcloud +> help compute`) is not sufficient for leaf-level syntax validation. +> Validation must occur at the specific leaf subcommand level. +> +> 4. **FORBIDDEN Web Search Fallback**: NEVER use `search_web`, web search, or +> external documentation search tools for gcloud CLI syntax. `gcloud help +> ` is the **EXCLUSIVE** authorized authority for command +> syntax. +> +> 5. **User Flag & Project Preservation**: When proposing intermediate command +> steps, **ALWAYS** preserve all user-specified flags (including +> `--project=`) in the proposed response text. +> +> 6. **Mandatory Plan Template**: When generating a plan, the response **MUST** +> copy this exact 4-step structure: +> +> - **Step 1**: Syntax Validation via `gcloud help ` +> - **Step 2**: Parameter Verification (confirming required and optional +> flags, and explicitly checking if the `--dry-run` or `--validate-only` +> flag is supported) +> - **Step 3**: Dry-Run Command Proposal (If `--dry-run` or +> `--validate-only` is supported, there MUST be a `--dry-run` or +> `--validate-only` invocation before the next step.) +> - **Step 4**: Command Proposal & Authorization (If the command is on the +> "Prohibited Operations" denylist, state that autonomous execution is +> forbidden, and the user MUST be explicitly asked for authorization to +> proceed. If the command is NOT on the denylist, propose or proceed +> with execution, while following *ALL* "Execution Constraints" below.) + This document provides essential guidelines and best practices for AI agents interacting with the Google Cloud SDK (`gcloud` CLI). Following these rules is critical to avoid hallucinated commands, flags, flag values, and positional argument syntax, prevent destructive actions, and minimize context window usage. -## Getting Started - -### 1. Installation - -If the `gcloud` executable is missing, refer to the official -[Google Cloud CLI Installation Guide](https://docs.cloud.google.com/sdk/docs/install-sdk.md.txt) -to install it on your platform (Linux, macOS, Windows, etc.). - -### 2. Authorization - -Authenticate the CLI with Google Cloud. Choose the flow that matches your -running environment: - -* **User Account (Interactive)**: Run `gcloud auth login`. Follow the browser - prompts to sign in. -* **User Account (Headless Flow)**: If operating on a terminal without a web - browser (e.g. containers, remote SSH), append the `--no-browser` flag: - `gcloud auth login --no-browser`. Copy the URL, sign in on another machine, - and return the authentication code. -* **Application Default Credentials (ADC)**: To authenticate code calls from - local applications or SDK libraries, set up ADC via `gcloud auth - application-default login` (append `--no-browser` for headless - environments). -* **Service Account (Best for Detached/Headless Automation)**: Authenticate - directly using a JSON key file. Ideal for fully automated, background tasks - and pipelines: `gcloud auth activate-service-account - --key-file=path/to/key.json`. Note that some organizations may restrict - access to JSON key files for security reasons. -* **Service Account Impersonation (Preferred for Local Pair-Programming - Agents)**: Leverage the human developer's existing user credentials to - assume a service account identity. Best for local development assistants to - avoid insecure private keys on human workstations: `gcloud config set - auth/impersonate_service_account SERVICE_ACCT_EMAIL` - -*Separation of Privilege (Critical)*: Both service account approaches ensure the -agent's permissions remain strictly distinct from the human user's wide access -limits (enforcing least privilege), and ensure actions are properly audited -under the agent's focused identity. *(Impersonation requires -`roles/iam.serviceAccountTokenCreator`)*. - -For more detailed strategies and authentication types (such as Workload Identity -Federation), see -[Authorizing the gcloud CLI](https://docs.cloud.google.com/sdk/docs/authorizing.md.txt). +## Execution Modes + +AI agents can interact with Google Cloud resources in two primary ways: + +- **Direct CLI Execution**: Executing `gcloud` commands directly in a local or + automated shell environment. See [CLI Usage](references/cli-usage.md) for + installation, authentication flows, and configuration management. +- **Model Context Protocol (MCP)**: Invoking structured tools via the Cloud + CLI remote MCP server (`run_gcloud_command`). See + [MCP Usage](references/mcp-usage.md) for tool schemas, parameter rules, and + server configuration. ## Core Principles ### 1. Explicit Command Validation (Mandatory) -Your internal knowledge of `gcloud` may be stale or prone to hallucination -(e.g., hallucinating commands, flags, flag values, or positional argument -syntax). You are **FORBIDDEN** from executing commands until you have validated -the exact syntax at the leaf level. - -* **Action**: Always call `gcloud help ` for the *exact* command you - intend to run (e.g., `gcloud help compute instances create`). +* **Action**: **ALWAYS** call `gcloud help ` for the *exact* command + that is intended to be run (e.g., `gcloud help compute instances create`). * **Verify**: Ensure the command, flags, flag values, and positional argument - syntax are valid for that specific leaf command before attempting execution. - Validation is not transitive from parent groups. + syntax are valid for that specific leaf command before attempting execution + or presenting plans. Validation is not transitive from parent groups. -### 2. Data Reduction Strategies +### 2. Data Reduction Strategies (Mandatory) -To save context window space and reduce latency, always minimize the volume of -data returned by `gcloud`. +Minimize the volume of data returned by `gcloud` to save context window space +and reduce latency. DO NOT execute any `list` command without including at least +one data reduction flag (`--limit`, `--filter`, or `--format`). -* **Projection**: Use `--format=json(key1, key2, ...)` to select only the - specific fields needed for your task. To understand the advanced projection +* **Projection**: Use `--format="json(key1, key2, ...)"` to select only the + specific fields needed for the task. To understand the advanced projection and formatting syntax, refer to `gcloud topic projections` and `gcloud topic formats`. @@ -96,9 +113,9 @@ data returned by `gcloud`. characters. To study the filter expression syntax, refer to `gcloud topic filters`. -* **Schema Discovery**: Unconstrained resource lists can quickly exhaust your +* **Schema Discovery**: Unconstrained resource lists can quickly exhaust the context window with redundant data. To prevent this, discover a resource's - schema before executing queries. If you are unsure of the JSON key path for + schema before executing queries. If unsure of the JSON key path for projecting fields (`--format`) or filtering (`--filter`), run the targeted resource's list command (if supported) with a single-item limit: @@ -116,15 +133,23 @@ data returned by `gcloud`. * **No Shell Operators**: Do not use command substitution (`$(...)`), pipes (`|`), or redirection (`>`, `>>`, `<`). This is to increase command safety and ensure commands are more easily understandable and reviewable by users. -* **No Interactivity**: Do not run interactive commands or commands requiring - a TTY (e.g., `gcloud interactive`). You must enforce non-interactive mode by - appending `--quiet` (or `-q`) to your commands. This ensures that defaults - are used or errors are raised if input is required. +* **Non-Interactive Execution (`--quiet` / `-q`)**: Pass the `--quiet` (or + `-q`) global flag on all execution commands (e.g., `gcloud pubsub topics + delete temp-topic --quiet --project=test-project`). AI agents run in + headless, non-interactive environments without a TTY or `stdin` input + handler. Without `--quiet`, commands that prompt for user confirmation (such + as deleting resources, approving defaults, or selecting unspecified regions) + will pause execution indefinitely waiting for input, causing background task + timeouts. Including `--quiet` forces non-interactive mode, causing `gcloud` + to automatically accept safe default choices or fail immediately with an + explicit error if required parameters are missing. +* **No Blind Lists**: NEVER execute a `list` command without `--limit`, + `--filter`, or `--format`. ### 4. Project and Location Scoping (Critical) To ensure commands are deterministic, non-interactive, and target the correct -environment, you must explicitly manage project and location scoping. +environment, they must explicitly provide project and location scoping. * **Explicit Project Target**: Do not rely on active configuration defaults. Always append `--project=` to all resource-manipulating and @@ -132,13 +157,13 @@ environment, you must explicitly manage project and location scoping. accidental execution against the wrong project. * **Prevent Location Prompts**: Many Google Cloud resources are regional or - zonal. If you omit the location flag (e.g., `--region`, `--zone`, or + zonal. If the location flag is omitted (e.g., `--region`, `--zone`, or `--location`), `gcloud` will trigger an interactive prompt to select a zone/region. This violates the **No Interactivity** rule. Always provide explicit location flags if the command requires them. -* **Location Discovery**: If you do not know the correct region, zone, or - location for a service, run discovery commands first (remembering to limit +* **Location Discovery**: If the correct region, zone, or location for a + service is not known, run discovery commands first (remembering to limit results if there are many): * **Compute Engine (VMs, Networks)**: @@ -161,8 +186,8 @@ environment, you must explicitly manage project and location scoping. ### Prohibited Operations (Denylist) -You are **strictly prohibited** from executing the following commands -autonomously. These require explicit human-in-the-loop authorization: +NEVER execute the following commands autonomously. These require explicit +human-in-the-loop authorization: * **Any IAM policy, role, or binding modification** (Security): Risk of privilege escalation, administrative lockout, service disruption, or @@ -182,35 +207,38 @@ autonomously. These require explicit human-in-the-loop authorization: ### Execution Guidelines -* **Dry Run (Mandatory)**: You MUST invoke a command with `--dry-run` (or - equivalent) first if it exists, before executing the actual command, to - preview changes. +* **Dry Run (Mandatory)**: If the `--dry-run` or `--validate-only` flag (or + equivalent) is listed in the command help output, ALWAYS include the flag in + the proposed command or initial execution step. ALWAYS preview changes with + `--dry-run` or `--validate-only` prior to actual execution. * **Long Running Operations**: For commands that support it, the `--async` flag is highly recommended for long-running operations to avoid blocking the agentic flow. Note that not every command has an `--async` flag. For commands that return an operation ID (whether via `--async` or by default), - you are responsible for polling for completion if the operation status is - needed for the next step. + operation status must be polled for completion, if needed for the next step. + +* **Non-Interactive Flag (`--quiet`)**: Include `--quiet` (or `-q`) on all + proposed or executed commands to guarantee non-interactive execution without + waiting for TTY confirmation prompts. ## Structured Workflows ### Discovery Workflow -When asked to perform a task on a service you are not familiar with: - -1. You MUST invoke help on a command (e.g., `gcloud help `) before - invoking it. -2. If you do not know the exact command, traverse the command tree by invoking - help on a command group (e.g., `gcloud help compute`) to discover available - subcommands and groups. -3. **Schema Discovery**: If you need to filter or project fields from a list - command, but do not know the exact JSON keys, first run `gcloud - list --limit=1 --format=json` to safely discover the schema. - **Never** run a raw `list` command without scoping constraints (like - `--limit=1`), as unconstrained results will pollute and exhaust your context - window. -4. Execute with data reduction flags. +When asked to perform a task on a service that is unfamiliar: + +1. **Invoke Help**: Call `gcloud help ` on the target leaf command + prior to execution. +2. **Traverse Command Tree**: Run help on command groups (e.g., `gcloud help + compute` or `gcloud help`) to discover available subgroups and commands if + the exact command is unknown. +3. **Discover Schema**: Run `gcloud list --limit=1 + --format=json` to inspect JSON keys before constructing filters or + projections. DO NOT execute unconstrained `list` commands without scoping + flags (e.g., `--limit=1`) to prevent context window exhaustion. +4. **Enforce Data Reduction**: Include data reduction flags (`--limit`, + `--filter`, `--format`) on all command executions. ## Quick Reference / Cheat Sheet @@ -232,3 +260,13 @@ List Locations | `gcloud locations list --project=` Refer to the [gcloud CLI Scripting Guide](https://docs.cloud.google.com/sdk/docs/scripting-gcloud.md.txt) for guidance on using the gcloud CLI in automation. + +## Reference Directory + +- [CLI Usage](references/cli-usage.md): Platform installation, authentication + methods (interactive, headless, ADC, service account keys, impersonation), + and local configuration management. + +- [MCP Usage](references/mcp-usage.md): Using the Cloud CLI remote MCP + server (`run_gcloud_command`), project parameter scoping, input files, and + execution guidelines. diff --git a/external/develop/devops/gcloud/references/cli-usage.md b/external/develop/devops/gcloud/references/cli-usage.md new file mode 100644 index 0000000..6b8ab5b --- /dev/null +++ b/external/develop/devops/gcloud/references/cli-usage.md @@ -0,0 +1,153 @@ +# gcloud CLI Usage + +This document provides reference information for installing, authorizing, and +configuring the Google Cloud SDK (`gcloud` CLI) in local and automated +environments. + +## Installation + +If the `gcloud` binary is not installed in the execution environment, refer to +the authoritative +[Google Cloud CLI Installation Guide](https://docs.cloud.google.com/sdk/docs/install-sdk.md.txt) +for platform-specific installation instructions (Linux, macOS, Windows, package +managers, and container images). + +### Component Management + +The `gcloud components` command group manages optional CLI components (such as +additional tools, emulators, and language runtimes): + +- **List available components:** + + ```bash + gcloud components list + ``` + +- **Install a component:** + + ```bash + gcloud components install {component_id} --quiet + ``` + +- **Update all installed components:** + + ```bash + gcloud components update --quiet + ``` + +*(Note: If `gcloud` was installed via a system package manager like APT or DNF, +use the system package manager to install components instead of `gcloud +components install`.)* + +## Authorization & Authentication + +Authenticate the CLI with Google Cloud according to the operational environment: + +- **User Account (Interactive):** + + ```bash + gcloud auth login + ``` + + Follow the browser prompts to sign in and grant access. + +- **User Account (Headless Flow):** + + For environments without an accessible web browser (containers, remote SSH): + + ```bash + gcloud auth login --no-browser + ``` + + Copy the generated URL, open it on another machine to complete sign-in, and + paste the authorization code back into the terminal. + +- **Application Default Credentials (ADC):** + + Configures credentials for client libraries and local applications: + + ```bash + gcloud auth application-default login + ``` + + Append `--no-browser` in headless environments. + +- **Service Account Key (Headless Automation):** + + ```bash + gcloud auth activate-service-account --key-file=path/to/key.json + ``` + + *Security note: Restrict file permissions on JSON keys or prefer Workload + Identity / Impersonation.* + +- **Service Account Impersonation (Preferred for Development & Agents):** + + Allows a user identity to temporarily assume a service account identity + without storing long-lived private key files: + + ```bash + gcloud config set auth/impersonate_service_account {service_account_email} + ``` + + Requires the `roles/iam.serviceAccountTokenCreator` role on the target + service account. This enforces least privilege and ensures audited access + under the target identity. + +- **Workload Identity Federation:** + + For CI/CD and external compute environments (GitHub Actions, AWS, on-prem), + authenticate using federated tokens without managing service account keys. + See + [Authorizing the gcloud CLI](https://docs.cloud.google.com/sdk/docs/authorizing.md.txt). + +## Local Configuration Management + +The `gcloud config` command group manages local configuration settings, +profiles, and default properties. + +### Named Configurations + +Configurations allow maintaining multiple isolated sets of properties (e.g., +dev, staging, prod): + +- **Create a new configuration:** + + ```bash + gcloud config configurations create {config_name} + ``` + +- **List existing configurations:** + + ```bash + gcloud config configurations list + ``` + +- **Activate a configuration:** + + ```bash + gcloud config configurations activate {config_name} + ``` + +### Setting Common Properties + +Properties set default values for flags across `gcloud` invocations: + +- **Set active project:** + + ```bash + gcloud config set core/project {project_id} + ``` + +- **Set default compute region and zone:** + + ```bash + gcloud config set compute/region {region} + gcloud config set compute/zone {zone} + ``` + +- **View all active configuration properties:** + + ```bash + gcloud config list + ``` diff --git a/external/develop/devops/gcloud/references/mcp-usage.md b/external/develop/devops/gcloud/references/mcp-usage.md new file mode 100644 index 0000000..15d058e --- /dev/null +++ b/external/develop/devops/gcloud/references/mcp-usage.md @@ -0,0 +1,187 @@ +# Cloud CLI Remote MCP Server Usage + +Google Cloud resources can be managed via the Model Context Protocol (MCP), +allowing AI agents to interact with Google Cloud using structured tool calls +rather than directly executing local shell commands. + +MCP operations for `gcloud` are executed through the **Cloud CLI remote MCP +server** (backed by the Cloud CLI Execution API, `cloudcli.googleapis.com`). + +## Server Endpoint & Tool Overview + +- **Server Endpoint:** `https://cloudcli.googleapis.com/mcp` +- **Transport:** HTTP (JSON-RPC 2.0) +- **API Name:** Cloud CLI Execution API (`cloudcli.googleapis.com`) +- **Available Tool:** `run_gcloud_command` + +The `run_gcloud_command` tool executes a single `gcloud` command securely in a +managed remote environment on behalf of the user. + +## Client Configuration (`mcp_config.json`) + +To connect an MCP client (such as Jetski) to the remote Cloud CLI MCP server, +configure the server entry in `mcp_config.json` with `authProviderType` set to +`"google_credentials"`: + +```json +{ + "mcpServers": { + "gcloud-remote": { + "serverUrl": "https://cloudcli.googleapis.com/mcp", + "authProviderType": "google_credentials" + } + } +} +``` + +> [!IMPORTANT] Specifying `"authProviderType": "google_credentials"` is +> mandatory. It instructs the MCP client to attach Application Default +> Credentials (ADC) with the `https://www.googleapis.com/auth/cloud-platform` +> OAuth scope. Omitting this field will cause the client to send unauthenticated +> requests, resulting in `401 Unauthorized` errors. + +## Prerequisites & IAM Requirements + +Before using the Cloud CLI remote MCP server, the target project and calling +identity must satisfy two mandatory prerequisites: + +### 1. API Enablement + +The Cloud CLI Execution API (`cloudcli.googleapis.com`) must be enabled on the +target project. + +- **Via Google Cloud Console (No CLI required):** + + 1. Open the [Google Cloud Console](https://console.cloud.google.com/). + 2. Navigate to **APIs & Services** --> **Library**. + 3. Search for **Cloud CLI Execution API** (or open the + [Cloud CLI Execution API Library Page](https://console.cloud.google.com/apis/library/cloudcli.googleapis.com)). + 4. Select the target project from the project dropdown. + 5. Click **Enable**. + +- **Via `gcloud` CLI:** + + ```bash + gcloud services enable cloudcli.googleapis.com --project={project_id} + ``` + +### 2. IAM Roles & Permissions + +- **MCP Access Role:** The caller identity must hold the **MCP Tool User** + role (`roles/mcp.toolUser`, which grants the `mcp.tools.call` permission) on + the target project. +- **Downstream Resource Roles:** The caller identity must also hold standard + IAM permissions on the underlying resources being queried or modified (e.g., + `roles/compute.viewer`, `roles/run.developer`). + +> [!CAUTION] If either the Cloud CLI Execution API is not enabled or the caller +> lacks the `roles/mcp.toolUser` role, the endpoint returns **`403 Forbidden`** +> during both tool discovery (`tools/list`) and tool invocation (`tools/call`). + +## Tool Parameters + +Calls to `run_gcloud_command` accept the following parameters: + +- **`command`** (string, required): The full `gcloud` command line string to + execute (e.g., `"gcloud compute instances list --project={resource_project} + --format=json"`). +- **`project`** (string, required): The resource name of the Google Cloud + project hosting the Cloud CLI Execution API in the format + `"projects/{api_project}"` (e.g., `"projects/my-api-project"`). +- **`input_files`** (list of objects, optional): Files to provision in the + remote execution environment before running the command. Each item contains + a relative `path` and string `contents`. + +> [!IMPORTANT] **API Host Project vs. Resource Project Context:** +> +> - The top-level **`project`** parameter (`"projects/{api_project}"`) is used +> **strictly for quota, billing, and API enablement** of the +> `cloudcli.googleapis.com` API itself. It does NOT set the project context +> for the command being executed. +> - For **project-scoped commands**, you MUST explicitly include +> `--project={resource_project}` within the `command` string. The target +> `{resource_project}` does NOT have to be the project hosting the Cloud CLI +> Execution API. +> - For **non-project-scoped commands** (such as billing or organization +> queries), you MUST include `--billing-project={billing_project}` in the +> `command` string if the underlying API requires a quota project. + +### Example Invocations + +#### 1. Basic Command Execution + +```json +{ + "command": "gcloud compute instances list --project=my-resource-project --format=json", + "project": "projects/my-cloudcli-api-project" +} +``` + +#### 2. Command with Input Files + +```json +{ + "command": "gcloud run services replace service-config.yaml --region=us-central1 --project=my-resource-project", + "project": "projects/my-cloudcli-api-project", + "input_files": [ + { + "path": "service-config.yaml", + "contents": "apiVersion: serving.knative.dev/v1\nkind: Service\nmetadata:\n name: my-service\n..." + } + ] +} +``` + +## Response Structure + +The tool returns an execution response containing: + +- `exit_code`: Numeric exit status of the command execution. **This is the + primary and authoritative indicator of command success or failure.** +- `stdout`: Standard output stream from the command. +- `stderr`: Standard error stream from the command. +- `output_files`: Any files generated by the command. + +> [!NOTE] - **Exit Code Authority:** A command is successful if and only if +> `exit_code == 0`. A non-zero `exit_code` indicates failure. +> +> - **Informational `stderr` Output:** In `gcloud`, `stderr` frequently +> contains standard status messages, progress updates, and asynchronous +> tracking IDs (such as `--async` operation IDs) even when the command +> executes successfully (`exit_code == 0`). Agents MUST NOT assume a command +> failed merely because `stderr` is non-empty. +> - **Error Diagnosis:** If `exit_code != 0`, diagnostic error messages may +> appear in either `stderr` or `stdout`. Inspect both streams to understand +> the failure and formulate a correction. + +## Prohibited & Unsupported Commands + +The Cloud CLI remote MCP server operates in a sandboxed, non-interactive +environment. The following list shows a few example `gcloud` commands that +aren't supported (such as command groups that manage local machine +configuration, credentials, interactive shells, or metadata). This list is +non-exhaustive and subject to the addition or removal of commands without +notice: + +- `gcloud auth` (Local authentication & credential management) +- `gcloud config` (Local CLI configuration profiles and properties) +- `gcloud iam service-accounts` (Service account management) +- `gcloud init` (Interactive setup wizard) +- `gcloud survey` (User feedback & surveys) +- `gcloud compute ssh` / `gcloud app instances ssh` (Interactive SSH shells) + +## Safety & Execution Guidelines + +- **Mandatory User Consent for Mutations:** Destructive or state-changing + commands (such as `create`, `delete`, `update`, or `patch`) modify or + destroy GCP resources. These commands must NOT be invoked autonomously + unless the user has explicitly authorized the action. +- **Asynchronous Operations (`--async`):** For long-running operations (such + as creating VM instances, GKE clusters, or database instances), always + append the `--async` flag in the `command` string to avoid execution + timeouts. +- **Data Reduction & Formatting:** Use `--format=json`, `--filter`, and + `--limit` in the `command` string to constrain output volume and prevent + context window bloat. +- **Non-Interactive Execution (`--quiet`):** Include `--quiet` (or `-q`) on + commands that might otherwise prompt for interactive user confirmation. diff --git a/external/develop/devops/playwright-best-practices/SKILL.md b/external/develop/devops/playwright-best-practices/SKILL.md index 9e30124..0da7362 100644 --- a/external/develop/devops/playwright-best-practices/SKILL.md +++ b/external/develop/devops/playwright-best-practices/SKILL.md @@ -4,7 +4,7 @@ description: Use when writing Playwright tests, fixing flaky tests, debugging fa license: MIT metadata: author: currents.dev - version: "1.1" + version: "1.2" --- # Playwright Best Practices diff --git a/external/develop/devops/playwright-best-practices/advanced/authentication-flows.md b/external/develop/devops/playwright-best-practices/advanced/authentication-flows.md new file mode 100644 index 0000000..24ad08c --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/authentication-flows.md @@ -0,0 +1,360 @@ +# Complex Authentication Flow Patterns + +## Table of Contents + +1. [Email Verification Flows](#email-verification-flows) +2. [Password Reset](#password-reset) +3. [Session Timeout](#session-timeout) +4. [Remember Me Persistence](#remember-me-persistence) +5. [Logout Patterns](#logout-patterns) +6. [Tips](#tips) +7. [Related](#related) + +> **When to use**: Testing email verification, password reset, session timeout/expiration, or remember-me functionality. For basic auth setup (storage state, OAuth mocking, MFA, role-based access), see [authentication.md](authentication.md). + +--- + +## Email Verification Flows + +### Capturing Verification Tokens + +Intercept API responses to capture verification tokens for testing: + +```typescript +test('completes registration with email verification', async ({ page }) => { + let capturedToken = ''; + + await page.route('**/api/auth/register', async (route) => { + const response = await route.fetch(); + const body = await response.json(); + capturedToken = body.verificationToken; + await route.fulfill({ response }); + }); + + await page.goto('/register'); + await page.getByLabel('Name').fill('New User'); + await page.getByLabel('Email').fill('newuser@test.com'); + await page.getByLabel('Password', { exact: true }).fill('SecurePass!'); + await page.getByLabel('Confirm password').fill('SecurePass!'); + await page.getByRole('button', { name: 'Create account' }).click(); + + await expect(page.getByText('Check your inbox')).toBeVisible(); + + expect(capturedToken).toBeTruthy(); + await page.goto(`/verify?token=${capturedToken}`); + + await expect(page.getByText('Email confirmed')).toBeVisible(); +}); +``` + +### Fully Mocked Verification + +```typescript +test('verifies email with mocked endpoints', async ({ page }) => { + const mockToken = 'test-verification-abc123'; + + await page.route('**/api/auth/register', async (route) => { + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ message: 'Verification sent', verificationToken: mockToken }), + }); + }); + + await page.route(`**/api/auth/verify?token=${mockToken}`, async (route) => { + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ verified: true }), + }); + }); + + await page.goto('/register'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Password', { exact: true }).fill('Password123!'); + await page.getByRole('button', { name: 'Sign up' }).click(); + + await expect(page.getByText('Check your inbox')).toBeVisible(); + + await page.goto(`/verify?token=${mockToken}`); + await expect(page.getByText('Email confirmed')).toBeVisible(); +}); +``` + +--- + +## Password Reset + +### Complete Reset Flow + +```typescript +test('resets password through email link', async ({ page }) => { + let resetToken = ''; + + await page.route('**/api/auth/forgot-password', async (route) => { + const response = await route.fetch(); + const body = await response.json(); + resetToken = body.resetToken; + await route.fulfill({ response }); + }); + + await page.goto('/forgot-password'); + await page.getByLabel('Email').fill('user@test.com'); + await page.getByRole('button', { name: 'Send link' }).click(); + + await expect(page.getByText('Reset email sent')).toBeVisible(); + + expect(resetToken).toBeTruthy(); + await page.goto(`/reset-password?token=${resetToken}`); + + await page.getByLabel('New password', { exact: true }).fill('NewPassword456!'); + await page.getByLabel('Confirm password').fill('NewPassword456!'); + await page.getByRole('button', { name: 'Update password' }).click(); + + await expect(page.getByText('Password updated')).toBeVisible(); +}); +``` + +### Expired Token Handling + +```typescript +test('shows error for expired reset token', async ({ page }) => { + await page.goto('/reset-password?token=expired-token'); + + await page.getByLabel('New password', { exact: true }).fill('NewPass!'); + await page.getByLabel('Confirm password').fill('NewPass!'); + await page.getByRole('button', { name: 'Update password' }).click(); + + await expect(page.getByRole('alert')).toContainText(/expired|invalid/i); +}); +``` + +### Password Strength Validation + +```typescript +test('enforces password requirements on reset', async ({ page }) => { + await page.goto('/reset-password?token=valid-token'); + + await page.getByLabel('New password', { exact: true }).fill('weak'); + await page.getByLabel('Confirm password').fill('weak'); + await page.getByRole('button', { name: 'Update password' }).click(); + + await expect(page.getByText(/at least 8 characters/i)).toBeVisible(); +}); +``` + +--- + +## Session Timeout + +### Detecting Expired Sessions + +```typescript +test('redirects to signin after session expires', async ({ page, context }) => { + await page.goto('/signin'); + await page.getByLabel('Email').fill('user@test.com'); + await page.getByLabel('Password').fill('Password!'); + await page.getByRole('button', { name: 'Sign in' }).click(); + await expect(page).toHaveURL('/home'); + + const cookies = await context.cookies(); + const sessionCookie = cookies.find((c) => c.name.includes('session')); + + if (sessionCookie) { + await context.clearCookies({ name: sessionCookie.name }); + } + + await page.goto('/profile'); + await expect(page).toHaveURL(/\/signin/); + await expect(page.getByText(/session.*expired|sign in again/i)).toBeVisible(); +}); +``` + +### Session Extension Warning + +```typescript +test('shows warning before session expires', async ({ page }) => { + await page.route('**/api/auth/session', async (route) => { + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ valid: true, expiresIn: 60 }), + }); + }); + + await page.goto('/home'); + + await expect(page.getByText(/session.*expir/i)).toBeVisible({ timeout: 10000 }); + await expect(page.getByRole('button', { name: /extend|stay signed in/i })).toBeVisible(); +}); +``` + +### Session Extension Action + +```typescript +test('extends session when user clicks extend', async ({ page }) => { + let sessionExtended = false; + + await page.route('**/api/auth/session', async (route) => { + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ valid: true, expiresIn: 60 }), + }); + }); + + await page.route('**/api/auth/refresh', async (route) => { + sessionExtended = true; + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ valid: true, expiresIn: 3600 }), + }); + }); + + await page.goto('/home'); + + await expect(page.getByRole('button', { name: /extend|stay signed in/i })).toBeVisible({ + timeout: 10000, + }); + await page.getByRole('button', { name: /extend|stay signed in/i }).click(); + + expect(sessionExtended).toBe(true); + await expect(page.getByText(/session.*expir/i)).not.toBeVisible(); +}); +``` + +--- + +## Remember Me Persistence + +### Persistent Session + +```typescript +test('persists session with remember me enabled', async ({ browser }) => { + const ctx1 = await browser.newContext(); + const page1 = await ctx1.newPage(); + + await page1.goto('/signin'); + await page1.getByLabel('Email').fill('user@test.com'); + await page1.getByLabel('Password').fill('Password!'); + await page1.getByLabel('Keep me signed in').check(); + await page1.getByRole('button', { name: 'Sign in' }).click(); + + await expect(page1).toHaveURL('/home'); + + const state = await ctx1.storageState(); + await ctx1.close(); + + const ctx2 = await browser.newContext({ storageState: state }); + const page2 = await ctx2.newPage(); + + await page2.goto('/home'); + await expect(page2).toHaveURL('/home'); + await expect(page2.getByText('Welcome')).toBeVisible(); + + await ctx2.close(); +}); +``` + +### Session-Only Login + +```typescript +test('session-only login does not persist across browser restarts', async ({ browser }) => { + const ctx1 = await browser.newContext(); + const page1 = await ctx1.newPage(); + + await page1.goto('/signin'); + await page1.getByLabel('Email').fill('user@test.com'); + await page1.getByLabel('Password').fill('Password!'); + // Leave "Remember me" unchecked + await expect(page1.getByLabel('Keep me signed in')).not.toBeChecked(); + await page1.getByRole('button', { name: 'Sign in' }).click(); + + await expect(page1).toHaveURL('/home'); + + // Only keep persistent cookies (filter out session cookies) + const cookies = await ctx1.cookies(); + await ctx1.close(); + + const persistentCookies = cookies.filter((c) => c.expires > 0); + const ctx2 = await browser.newContext(); + await ctx2.addCookies(persistentCookies); + const page2 = await ctx2.newPage(); + + await page2.goto('/home'); + + // Should redirect to login since session was not persisted + await expect(page2).toHaveURL(/\/signin/); + + await ctx2.close(); +}); +``` + +--- + +## Logout Patterns + +### Standard Logout with Session Cleanup + +```typescript +test.use({ storageState: '.auth/user.json' }); + +test('logs out and clears session', async ({ page, context }) => { + await page.goto('/home'); + + await page.getByRole('button', { name: /account|menu/i }).click(); + await page.getByRole('menuitem', { name: 'Sign out' }).click(); + + await expect(page).toHaveURL('/signin'); + + const cookies = await context.cookies(); + const sessionCookies = cookies.filter((c) => c.name.includes('session') || c.name.includes('token')); + expect(sessionCookies).toHaveLength(0); + + await page.goto('/home'); + await expect(page).toHaveURL(/\/signin/); +}); +``` + +### Logout from All Devices + +```typescript +test('logs out from all devices', async ({ page }) => { + let logoutAllCalled = false; + + await page.route('**/api/auth/logout-all', async (route) => { + logoutAllCalled = true; + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ message: 'Logged out everywhere' }), + }); + }); + + await page.goto('/settings/security'); + + await page.getByRole('button', { name: 'Sign out everywhere' }).click(); + await page.getByRole('dialog').getByRole('button', { name: 'Confirm' }).click(); + + expect(logoutAllCalled).toBe(true); + await expect(page).toHaveURL(/\/signin/); +}); +``` + +--- + +## Tips + +1. **Configure shorter session timeouts in test environments** — Enables testing timeout behavior without slow tests +2. **Test token expiration edge cases** — Expired tokens, invalid tokens, already-used tokens +3. **Verify cleanup on logout** — Check both cookies and localStorage are cleared +4. **Test the full flow end-to-end** — Password reset should verify login with new password works + +--- + +## Related + +- [authentication.md](authentication.md) — Storage state, OAuth mocking, MFA, role-based access, API login +- [fixtures-hooks.md](../core/fixtures-hooks.md) — Creating auth fixtures +- [third-party.md](./third-party.md) — Mocking external auth providers diff --git a/external/develop/devops/playwright-best-practices/advanced/authentication.md b/external/develop/devops/playwright-best-practices/advanced/authentication.md new file mode 100644 index 0000000..02c2dd7 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/authentication.md @@ -0,0 +1,871 @@ +# Authentication Testing + +## Table of Contents + +1. [Quick Reference](#quick-reference) +2. [Patterns](#patterns) +3. [Decision Guide](#decision-guide) +4. [Anti-Patterns](#anti-patterns) +5. [Troubleshooting](#troubleshooting) +6. [Related](#related) + +> **When to use**: Apps with login, session management, or protected routes. Authentication is the most common source of slow test suites. + +## Quick Reference + +```typescript +// Storage state reuse — the #1 pattern for fast auth +await page.goto("/login"); +await page.getByLabel("Username").fill("testuser@example.com"); +await page.getByLabel("Password").fill("secretPass123"); +await page.getByRole("button", { name: "Log in" }).click(); +await page.context().storageState({ path: ".auth/session.json" }); + +// Reuse in config — every test starts authenticated +{ + use: { + storageState: ".auth/session.json" + } +} + +// API login — skip the UI entirely +const context = await browser.newContext(); +const response = await context.request.post("/api/auth/login", { + data: { email: "testuser@example.com", password: "secretPass123" }, +}); +await context.storageState({ path: ".auth/session.json" }); +``` + +## Patterns + +### Storage State Reuse + +**Use when**: You need authenticated tests and want to avoid logging in before every test. +**Avoid when**: Tests require completely fresh sessions, or you are testing the login flow itself. + +`storageState` serializes cookies and localStorage to a JSON file. Load it in any browser context to start authenticated instantly. + +```typescript +// scripts/generate-auth.ts — run once to generate the state file +import { chromium } from "@playwright/test"; + +async function generateAuthState() { + const browser = await chromium.launch(); + const context = await browser.newContext(); + const page = await context.newPage(); + + await page.goto("http://localhost:4000/login"); + await page.getByLabel("Username").fill("testuser@example.com"); + await page.getByLabel("Password").fill("secretPass123"); + await page.getByRole("button", { name: "Log in" }).click(); + await page.waitForURL("/home"); + + await context.storageState({ path: ".auth/session.json" }); + await browser.close(); +} + +generateAuthState(); +``` + +```typescript +// playwright.config.ts — load saved state for all tests +import { defineConfig } from "@playwright/test"; + +export default defineConfig({ + use: { + baseURL: "http://localhost:4000", + storageState: ".auth/session.json", + }, +}); +``` + +```typescript +// tests/home.spec.ts — test starts already logged in +import { test, expect } from "@playwright/test"; + +test("authenticated user sees home page", async ({ page }) => { + await page.goto("/home"); + await expect(page.getByRole("heading", { name: "Home" })).toBeVisible(); +}); +``` + +### Global Setup Authentication + +**Use when**: You want to authenticate once before the entire test suite runs. +**Avoid when**: Different tests need different users, or your tokens expire faster than your suite runs. + +```typescript +// global-setup.ts +import { chromium, type FullConfig } from "@playwright/test"; + +async function globalSetup(config: FullConfig) { + const { baseURL } = config.projects[0].use; + const browser = await chromium.launch(); + const context = await browser.newContext(); + const page = await context.newPage(); + + await page.goto(`${baseURL}/login`); + await page.getByLabel("Username").fill(process.env.TEST_USER_EMAIL!); + await page.getByLabel("Password").fill(process.env.TEST_USER_PASSWORD!); + await page.getByRole("button", { name: "Log in" }).click(); + await page.waitForURL("**/home"); + + await context.storageState({ path: ".auth/session.json" }); + await browser.close(); +} + +export default globalSetup; +``` + +```typescript +// playwright.config.ts +import { defineConfig } from "@playwright/test"; + +export default defineConfig({ + globalSetup: require.resolve("./global-setup"), + use: { + baseURL: "http://localhost:4000", + storageState: ".auth/session.json", + }, +}); +``` + +Add `.auth/` to `.gitignore`. Auth state files contain session tokens and should never be committed. + +### Per-Worker Authentication + +**Use when**: Each parallel worker needs its own authenticated session to avoid race conditions for tests that modify server-side state. +**Avoid when**: Tests are read-only and a modifying shared session is safe, you can use a single shared account. + +> **Sharded runs**: `parallelIndex` resets per shard, so different shards can have workers with the same index. To avoid collisions, include the shard identifier in the username (e.g., `worker-${SHARD_INDEX}-${parallelIndex}@example.com`) by passing a `SHARD_INDEX` environment variable from your CI matrix. + +```typescript +// fixtures/auth.ts +import { test as base, type BrowserContext } from "@playwright/test"; + +type AuthFixtures = { + authenticatedContext: BrowserContext; +}; + +export const test = base.extend<{}, AuthFixtures>({ + authenticatedContext: [ + async ({ browser }, use) => { + const context = await browser.newContext(); + const page = await context.newPage(); + + await page.goto("/login"); + await page + .getByLabel("Username") + .fill(`worker-${test.info().parallelIndex}@example.com`); + await page.getByLabel("Password").fill("secretPass123"); + await page.getByRole("button", { name: "Log in" }).click(); + await page.waitForURL("/home"); + await page.close(); + + await use(context); + await context.close(); + }, + { scope: "worker" }, + ], +}); + +export { expect } from "@playwright/test"; +``` + +```typescript +// tests/settings.spec.ts +import { test, expect } from "../fixtures/auth"; + +test("update display name", async ({ authenticatedContext }) => { + const page = await authenticatedContext.newPage(); + await page.goto("/settings/profile"); + await page.getByLabel("Display name").fill("Updated Name"); + await page.getByRole("button", { name: "Save" }).click(); + await expect(page.getByText("Profile saved")).toBeVisible(); +}); +``` + +### Multiple Roles + +**Use when**: Your app has role-based access control and you need to test different permission levels. +**Avoid when**: Your app has a single user role. + +```typescript +// global-setup.ts — authenticate all roles +import { chromium, type FullConfig } from "@playwright/test"; + +const accounts = [ + { + role: "admin", + email: "admin@example.com", + password: process.env.ADMIN_PASSWORD!, + }, + { + role: "member", + email: "member@example.com", + password: process.env.MEMBER_PASSWORD!, + }, + { + role: "guest", + email: "guest@example.com", + password: process.env.GUEST_PASSWORD!, + }, +]; + +async function globalSetup(config: FullConfig) { + const { baseURL } = config.projects[0].use; + + for (const { role, email, password } of accounts) { + const browser = await chromium.launch(); + const context = await browser.newContext(); + const page = await context.newPage(); + + await page.goto(`${baseURL}/login`); + await page.getByLabel("Username").fill(email); + await page.getByLabel("Password").fill(password); + await page.getByRole("button", { name: "Log in" }).click(); + await page.waitForURL("**/home"); + + await context.storageState({ path: `.auth/${role}.json` }); + await browser.close(); + } +} + +export default globalSetup; +``` + +```typescript +// playwright.config.ts — one project per role +import { defineConfig } from "@playwright/test"; + +export default defineConfig({ + globalSetup: require.resolve("./global-setup"), + projects: [ + { + name: "admin", + use: { storageState: ".auth/admin.json" }, + testMatch: "**/*.admin.spec.ts", + }, + { + name: "member", + use: { storageState: ".auth/member.json" }, + testMatch: "**/*.member.spec.ts", + }, + { + name: "guest", + use: { storageState: ".auth/guest.json" }, + testMatch: "**/*.guest.spec.ts", + }, + { + name: "anonymous", + use: { storageState: { cookies: [], origins: [] } }, + testMatch: "**/*.anon.spec.ts", + }, + ], +}); +``` + +```typescript +// tests/admin-panel.admin.spec.ts +import { test, expect } from "@playwright/test"; + +test("admin can access user management", async ({ page }) => { + await page.goto("/admin/users"); + await expect( + page.getByRole("heading", { name: "User Management" }) + ).toBeVisible(); + await expect(page.getByRole("button", { name: "Remove user" })).toBeEnabled(); +}); +``` + +```typescript +// tests/admin-panel.guest.spec.ts +import { test, expect } from "@playwright/test"; + +test("guest cannot access admin panel", async ({ page }) => { + await page.goto("/admin/users"); + await expect(page.getByText("Access denied")).toBeVisible(); +}); +``` + +**Alternative**: Use a fixture that accepts a role parameter when you need role switching within a single spec file. + +```typescript +// fixtures/auth.ts — role-based fixture +import { test as base, type Page } from "@playwright/test"; +import fs from "fs"; + +type RoleFixtures = { + loginAs: (role: "admin" | "member" | "guest") => Promise; +}; + +export const test = base.extend({ + loginAs: async ({ browser }, use) => { + const pages: Page[] = []; + + await use(async (role) => { + const statePath = `.auth/${role}.json`; + if (!fs.existsSync(statePath)) { + throw new Error( + `Auth state for role "${role}" not found at ${statePath}` + ); + } + const context = await browser.newContext({ storageState: statePath }); + const page = await context.newPage(); + pages.push(page); + return page; + }); + + for (const page of pages) { + await page.context().close(); + } + }, +}); + +export { expect } from "@playwright/test"; +``` + +```typescript +// tests/role-comparison.spec.ts +import { test, expect } from "../fixtures/auth"; + +test("admin sees remove button, guest does not", async ({ loginAs }) => { + const adminPage = await loginAs("admin"); + await adminPage.goto("/admin/users"); + await expect( + adminPage.getByRole("button", { name: "Remove user" }) + ).toBeVisible(); + + const guestPage = await loginAs("guest"); + await guestPage.goto("/admin/users"); + await expect(guestPage.getByText("Access denied")).toBeVisible(); +}); +``` + +### OAuth/SSO Mocking + +**Use when**: Your app authenticates via a third-party OAuth provider and you cannot hit the real provider in tests. +**Avoid when**: You have a dedicated test tenant on the OAuth provider. + +A typical OAuth flow works like this: + +1. User clicks "Sign in with Provider" → browser navigates to `https://accounts.provider.com/authorize?...` +2. User authenticates on the provider's page → provider redirects back to your app's **callback route** (e.g. `http://localhost:4000/auth/callback?code=ABC&state=XYZ`) +3. Your backend exchanges the `code` for an access token, creates a session, and redirects the user to a logged-in page + +In tests you can short-circuit step 2 with `page.route()`: intercept the outbound request to the provider and respond with a `302` redirect straight to your callback route, supplying a mock `code` and `state`. Your backend still executes its normal callback handler — the only part that's mocked is the provider's authorization page. + +For cases where you want to skip the browser redirect entirely, a second approach calls a **test-only API endpoint** that creates the session server-side and returns the session cookie directly. + +```typescript +// tests/oauth-login.spec.ts — mock the callback route +import { test, expect } from "@playwright/test"; + +test("login via mocked OAuth flow", async ({ page }) => { + await page.route("https://accounts.provider.com/**", async (route) => { + const callbackUrl = new URL("http://localhost:4000/auth/callback"); + callbackUrl.searchParams.set("code", "mock-auth-code-xyz"); + callbackUrl.searchParams.set("state", "expected-state-value"); + await route.fulfill({ + status: 302, + headers: { location: callbackUrl.toString() }, + }); + }); + + await page.goto("/login"); + await page.getByRole("button", { name: "Sign in with Provider" }).click(); + + await page.waitForURL("/home"); + await expect(page.getByRole("heading", { name: "Home" })).toBeVisible(); +}); +``` + +```typescript +// tests/oauth-login.spec.ts — API-based session injection +import { test, expect } from "@playwright/test"; + +test("bypass OAuth entirely via API session injection", async ({ + page, +}) => { + // Call a test-only endpoint that creates a session without OAuth + const response = await page.request.post("/api/test/create-session", { + data: { + email: "oauth-user@example.com", + provider: "provider", + role: "member", + }, + }); + expect(response.ok()).toBeTruthy(); + + await page.context().storageState({ path: ".auth/oauth-user.json" }); + await page.goto("/home"); + await expect(page.getByRole("heading", { name: "Home" })).toBeVisible(); +}); +``` + +**Backend requirement**: Your backend must expose a test-only session creation endpoint (guarded by `NODE_ENV=test`) or accept a known test OAuth code. + +### MFA Handling + +**Use when**: Your app requires two-factor authentication (TOTP, SMS, email codes). +**Avoid when**: MFA is optional and you can disable it for test accounts. + +**Strategy 1**: Generate real TOTP codes from a shared secret. + +```typescript +// helpers/totp.ts +import * as OTPAuth from "otpauth"; + +export function generateTOTP(secret: string): string { + const totp = new OTPAuth.TOTP({ + secret: OTPAuth.Secret.fromBase32(secret), + digits: 6, + period: 30, + algorithm: "SHA1", + }); + return totp.generate(); +} +``` + +```typescript +// tests/mfa-login.spec.ts +import { test, expect } from "@playwright/test"; +import { generateTOTP } from "../helpers/totp"; + +test("login with TOTP two-factor auth", async ({ page }) => { + await page.goto("/login"); + await page.getByLabel("Username").fill("mfa-user@example.com"); + await page.getByLabel("Password").fill("secretPass123"); + await page.getByRole("button", { name: "Log in" }).click(); + + await expect(page.getByText("Enter your authentication code")).toBeVisible(); + + const code = generateTOTP(process.env.MFA_TOTP_SECRET!); + await page.getByLabel("Authentication code").fill(code); + await page.getByRole("button", { name: "Verify" }).click(); + + await page.waitForURL("/home"); + await expect(page.getByRole("heading", { name: "Home" })).toBeVisible(); +}); +``` + +**Strategy 2**: Mock MFA at the backend level. Have your backend accept a known bypass code (e.g., `000000`) when `NODE_ENV=test`. + +**Strategy 3**: Disable MFA for test accounts at the infrastructure level. + +### Session Refresh + +**Use when**: Your tokens expire during long test runs. +**Avoid when**: Your test suite runs quickly and tokens outlast the entire run. + +```typescript +// fixtures/auth-with-refresh.ts +import { test as base, type BrowserContext } from "@playwright/test"; +import fs from "fs"; + +type AuthFixtures = { + authenticatedPage: import("@playwright/test").Page; +}; + +export const test = base.extend({ + authenticatedPage: async ({ browser }, use) => { + const statePath = ".auth/session.json"; + + let context: BrowserContext; + if (fs.existsSync(statePath)) { + context = await browser.newContext({ storageState: statePath }); + const page = await context.newPage(); + + const response = await page.request.get("/api/auth/me"); + if (response.ok()) { + await use(page); + await context.close(); + return; + } + await context.close(); + } + + context = await browser.newContext(); + const page = await context.newPage(); + await page.goto("/login"); + await page.getByLabel("Username").fill(process.env.TEST_USER_EMAIL!); + await page.getByLabel("Password").fill(process.env.TEST_USER_PASSWORD!); + await page.getByRole("button", { name: "Log in" }).click(); + await page.waitForURL("/home"); + + await context.storageState({ path: statePath }); + + await use(page); + await context.close(); + }, +}); + +export { expect } from "@playwright/test"; +``` + +### Login Page Object + +**Use when**: Multiple test files need to log in and you want consistent, maintainable login logic. +**Avoid when**: You use `storageState` everywhere and never navigate through the login UI in tests. + +```typescript +// page-objects/LoginPage.ts +import { type Page, type Locator, expect } from "@playwright/test"; + +export class LoginPage { + readonly page: Page; + readonly usernameInput: Locator; + readonly passwordInput: Locator; + readonly loginButton: Locator; + readonly errorMessage: Locator; + readonly forgotPasswordLink: Locator; + + constructor(page: Page) { + this.page = page; + this.usernameInput = page.getByLabel("Username"); + this.passwordInput = page.getByLabel("Password"); + this.loginButton = page.getByRole("button", { name: "Log in" }); + this.errorMessage = page.getByRole("alert"); + this.forgotPasswordLink = page.getByRole("link", { + name: "Forgot password", + }); + } + + async goto() { + await this.page.goto("/login"); + await expect(this.loginButton).toBeVisible(); + } + + async login(username: string, password: string) { + await this.usernameInput.fill(username); + await this.passwordInput.fill(password); + await this.loginButton.click(); + } + + async loginAndWaitForHome(username: string, password: string) { + await this.login(username, password); + await this.page.waitForURL("/home"); + } + + async expectError(message: string | RegExp) { + await expect(this.errorMessage).toContainText(message); + } + + async expectFieldError(field: "username" | "password", message: string) { + const input = + field === "username" ? this.usernameInput : this.passwordInput; + await expect(input).toHaveAttribute("aria-invalid", "true"); + const errorId = await input.getAttribute("aria-describedby"); + if (errorId) { + await expect(this.page.locator(`#${errorId}`)).toContainText(message); + } + } +} +``` + +```typescript +// tests/login.spec.ts +import { test, expect } from "@playwright/test"; +import { LoginPage } from "../page-objects/LoginPage"; + +test.use({ storageState: { cookies: [], origins: [] } }); + +test.describe("login page", () => { + let loginPage: LoginPage; + + test.beforeEach(async ({ page }) => { + loginPage = new LoginPage(page); + await loginPage.goto(); + }); + + test("successful login redirects to home", async ({ page }) => { + await loginPage.loginAndWaitForHome( + "testuser@example.com", + "secretPass123" + ); + await expect(page.getByRole("heading", { name: "Home" })).toBeVisible(); + }); + + test("wrong password shows error", async () => { + await loginPage.login("testuser@example.com", "wrong-password"); + await loginPage.expectError("Invalid username or password"); + }); + + test("empty fields show validation errors", async () => { + await loginPage.loginButton.click(); + await loginPage.expectFieldError("username", "Username is required"); + }); + + test("forgot password link navigates correctly", async ({ page }) => { + await loginPage.forgotPasswordLink.click(); + await page.waitForURL("/forgot-password"); + await expect( + page.getByRole("heading", { name: "Reset password" }) + ).toBeVisible(); + }); +}); +``` + +### API-Based Login + +**Use when**: You want the fastest possible authentication without any browser interaction. +**Avoid when**: You are specifically testing the login UI. + +API login is typically 5-10x faster than UI login. + +```typescript +// global-setup.ts — API-based login (fastest) +import { request, type FullConfig } from "@playwright/test"; + +async function globalSetup(config: FullConfig) { + const { baseURL } = config.projects[0].use; + + const requestContext = await request.newContext({ baseURL }); + + const response = await requestContext.post("/api/auth/login", { + data: { + email: process.env.TEST_USER_EMAIL!, + password: process.env.TEST_USER_PASSWORD!, + }, + }); + + if (!response.ok()) { + throw new Error( + `API login failed: ${response.status()} ${await response.text()}` + ); + } + + await requestContext.storageState({ path: ".auth/session.json" }); + await requestContext.dispose(); +} + +export default globalSetup; +``` + +```typescript +// fixtures/api-auth.ts — fixture version for per-test authentication +import { test as base } from "@playwright/test"; + +export const test = base.extend({ + authenticatedPage: async ({ browser, playwright }, use) => { + const apiContext = await playwright.request.newContext({ + baseURL: "http://localhost:4000", + }); + + await apiContext.post("/api/auth/login", { + data: { + email: "testuser@example.com", + password: "secretPass123", + }, + }); + + const state = await apiContext.storageState(); + const context = await browser.newContext({ storageState: state }); + const page = await context.newPage(); + + await use(page); + + await context.close(); + await apiContext.dispose(); + }, +}); + +export { expect } from "@playwright/test"; +``` + +### Unauthenticated Tests + +**Use when**: Testing the login page, signup flow, password reset, public pages, or redirect behavior for unauthenticated users. +**Avoid when**: The test requires a logged-in user. + +When your config sets a default `storageState`, you must explicitly clear it for unauthenticated tests. + +```typescript +// tests/public-pages.spec.ts +import { test, expect } from "@playwright/test"; + +test.use({ storageState: { cookies: [], origins: [] } }); + +test.describe("unauthenticated access", () => { + test("homepage is accessible without login", async ({ page }) => { + await page.goto("/"); + await expect(page.getByRole("heading", { name: "Welcome" })).toBeVisible(); + await expect(page.getByRole("link", { name: "Log in" })).toBeVisible(); + }); + + test("protected route redirects to login", async ({ page }) => { + await page.goto("/home"); + await page.waitForURL("**/login**"); + expect(page.url()).toContain("redirect=%2Fhome"); + }); + + test("expired session shows re-login prompt", async ({ page, context }) => { + await page.goto("/home"); + await context.clearCookies(); + + await page.goto("/settings"); + await page.waitForURL("**/login**"); + await expect(page.getByText("Your session has expired")).toBeVisible(); + }); + + test("signup flow creates account", async ({ page }) => { + await page.goto("/signup"); + await page.getByLabel("Name").fill("New User"); + await page.getByLabel("Email").fill(`test-${Date.now()}@example.com`); + await page.getByLabel("Password", { exact: true }).fill("secretPass123"); + await page.getByLabel("Confirm password").fill("secretPass123"); + await page.getByRole("button", { name: "Create account" }).click(); + + await page.waitForURL("/onboarding"); + await expect(page.getByText("Welcome, New User")).toBeVisible(); + }); +}); +``` + +## Decision Guide + +| Scenario | Approach | Speed | Isolation | When to Choose | +| -------------------------------- | ------------------------------ | -------- | -------------- | -------------------------------------------------------------- | +| Most tests need auth | Global setup + `storageState` | Fastest | Shared session | Default for nearly every project | +| Tests modify user state | Per-worker fixture | Fast | Per worker | Tests update profile, change settings, or mutate data | +| Multiple user roles | Per-project `storageState` | Fastest | Per role | App has admin/member/guest roles | +| Testing the login page | No `storageState` | N/A | Full | Use `test.use({ storageState: { cookies: [], origins: [] } })` | +| OAuth/SSO provider | Mock the callback | Fast | Per test | Never hit real OAuth providers in CI | +| MFA is required | TOTP generation or bypass | Moderate | Per test | Generate real TOTP codes or use a test-mode bypass | +| Token expires mid-suite | Session refresh fixture | Fast | Per check | Fixture validates the session before use | +| Single test needs different user | `loginAs(role)` fixture | Moderate | Per call | Rare: prefer per-project roles | +| API-first app (no login UI) | API login via `request.post()` | Fastest | Per test | No browser needed for auth | + +### UI Login vs API Login vs Storage State + +```text +Need to test the login page itself? +├── Yes → UI login with LoginPage POM, no storageState +└── No → Do you have a login API endpoint? + ├── Yes → API login in global setup, save storageState (fastest) + └── No → UI login in global setup, save storageState + └── Tokens expire quickly? + ├── Yes → Add session refresh fixture + └── No → Standard storageState reuse is fine +``` + +## Anti-Patterns + +| Don't Do This | Problem | Do This Instead | +| ------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------- | +| Log in via UI before every test | Adds 2-5 seconds per test | Use `storageState` to skip login entirely | +| Share a single auth state file across parallel workers that mutate state | Race conditions | Use per-worker fixtures with `{ scope: 'worker' }` | +| Hardcode credentials in test files | Security risk | Use environment variables and `.env` files | +| Ignore token expiration | Tests fail intermittently with 401 errors | Add a session validity check in your auth fixture | +| Hit real OAuth providers in CI | Flaky: rate limits, CAPTCHA, network issues | Mock the OAuth callback or use API session injection | +| Use `page.waitForTimeout(2000)` after login | Arbitrary delay | `await page.waitForURL('/home')` or `await expect(heading).toBeVisible()` | +| Store `.auth/*.json` files in git | Tokens in version control | Add `.auth/` to `.gitignore` | +| Create one "god" test account with all permissions | Cannot test role-based access control | Create separate accounts per role | +| Use `browser.newContext()` without `storageState` for authenticated tests | Every context starts unauthenticated | Pass `storageState` when creating the context | +| Test MFA by disabling it everywhere | You never test the MFA flow | Use TOTP generation for at least one test | + +## Troubleshooting + +### Global setup fails with "Target page, context or browser has been closed" + +**Cause**: The login page redirected unexpectedly, or the browser closed before `storageState()` was called. + +**Fix**: + +- Add `await page.waitForURL()` after the login action +- Check that `baseURL` in your config matches the actual server URL and protocol +- Add error handling to global setup: + +```typescript +const response = await page.waitForResponse("**/api/auth/**"); +if (!response.ok()) { + throw new Error( + `Login failed in global setup: ${response.status()} ${await response.text()}` + ); +} +``` + +### Tests fail with 401 Unauthorized after running for a while + +**Cause**: The session token saved in `storageState` has expired. + +**Fix**: + +- Use the session refresh fixture pattern +- Increase token expiry in test environment configuration +- Switch to API-based login in a worker-scoped fixture + +### `storageState` file is empty or contains no cookies + +**Cause**: `storageState()` was called before the login response set cookies. + +**Fix**: + +- Wait for the post-login page to load: `await page.waitForURL('/home')` +- Verify cookies exist before saving: + +```typescript +const cookies = await context.cookies(); +if (cookies.length === 0) { + throw new Error("No cookies found after login"); +} +await context.storageState({ path: ".auth/session.json" }); +``` + +### Different browsers get different cookies + +**Cause**: Some auth flows set cookies with `SameSite=Strict` or use browser-specific cookie behavior. + +**Fix**: + +- Generate separate auth state files per browser project +- Check if your auth uses `SameSite=None; Secure` cookies that require HTTPS: + +```typescript +projects: [ + { + name: 'chromium', + use: { ...devices['Desktop Chrome'], storageState: '.auth/chromium-session.json' }, + }, + { + name: 'firefox', + use: { ...devices['Desktop Firefox'], storageState: '.auth/firefox-session.json' }, + }, +], +``` + +### Parallel tests interfere with each other's sessions + +**Cause**: Multiple workers share the same test account and one worker's actions affect others. + +**Fix**: + +- Use per-worker test accounts: `worker-${test.info().parallelIndex}@example.com` +- Use the per-worker authentication fixture pattern +- Make tests idempotent + +### OAuth mock does not work — still redirects to real provider + +**Cause**: `page.route()` was registered after the navigation that triggers the OAuth redirect. + +**Fix**: + +- Register route handlers before any navigation: call `page.route()` before `page.goto()` +- Log the actual redirect URL to verify the pattern: + +```typescript +page.on("request", (req) => { + if (req.url().includes("oauth") || req.url().includes("accounts.provider")) { + console.log("OAuth request:", req.url()); + } +}); +``` + +## Related + +- [fixtures-hooks.md](../core/fixtures-hooks.md) — custom fixtures for auth setup and teardown +- [configuration.md](../core/configuration.md) — `storageState`, projects, and global setup configuration +- [global-setup.md](../core/global-setup.md) — global setup patterns and project dependencies +- [network-advanced.md](network-advanced.md) — route interception patterns used in OAuth mocking +- [api-testing.md](../testing-patterns/api-testing.md) — API request context used in API-based login +- [flaky-tests.md](../debugging/flaky-tests.md) — diagnosing auth-related flakiness diff --git a/external/develop/devops/playwright-best-practices/advanced/clock-mocking.md b/external/develop/devops/playwright-best-practices/advanced/clock-mocking.md new file mode 100644 index 0000000..073d087 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/clock-mocking.md @@ -0,0 +1,364 @@ +# Date, Time & Clock Mocking + +## Table of Contents + +1. [Clock API Basics](#clock-api-basics) +2. [Fixed Time Testing](#fixed-time-testing) +3. [Time Advancement](#time-advancement) +4. [Timezone Testing](#timezone-testing) +5. [Timer Mocking](#timer-mocking) + +## Clock API Basics + +### Install Clock + +```typescript +test("mock current time", async ({ page }) => { + // Install clock before navigating + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + + await page.goto("/dashboard"); + + // Page sees January 15, 2025 as current date + await expect(page.getByText("January 15, 2025")).toBeVisible(); +}); +``` + +### Clock with Fixture + +```typescript +// fixtures/clock.fixture.ts +import { test as base } from "@playwright/test"; + +type ClockFixtures = { + mockTime: (date: Date | string) => Promise; +}; + +export const test = base.extend({ + mockTime: async ({ page }, use) => { + await use(async (date) => { + const time = typeof date === "string" ? new Date(date) : date; + await page.clock.install({ time }); + }); + }, +}); + +// Usage +test("subscription expiry", async ({ page, mockTime }) => { + await mockTime("2025-12-31T23:59:00"); + await page.goto("/subscription"); + + await expect(page.getByText("Expires today")).toBeVisible(); +}); +``` + +## Fixed Time Testing + +### Test Date-Dependent Features + +```typescript +test("show holiday banner in December", async ({ page }) => { + await page.clock.install({ time: new Date("2025-12-20T10:00:00") }); + + await page.goto("/"); + + await expect(page.getByRole("banner", { name: /holiday/i })).toBeVisible(); +}); + +test("no holiday banner in January", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T10:00:00") }); + + await page.goto("/"); + + await expect(page.getByRole("banner", { name: /holiday/i })).toBeHidden(); +}); +``` + +### Test Relative Time Display + +```typescript +test("shows relative time correctly", async ({ page }) => { + // Fix time to control "posted 2 hours ago" text + await page.clock.install({ time: new Date("2025-06-15T14:00:00") }); + + // Mock API to return post with known timestamp + await page.route("**/api/posts/1", (route) => + route.fulfill({ + json: { + id: 1, + title: "Test Post", + createdAt: "2025-06-15T12:00:00Z", // 2 hours before mock time + }, + }), + ); + + await page.goto("/posts/1"); + + await expect(page.getByText("2 hours ago")).toBeVisible(); +}); +``` + +### Test Date Boundaries + +```typescript +test.describe("end of month billing", () => { + test("shows billing on last day of month", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-31T10:00:00") }); + await page.goto("/billing"); + + await expect(page.getByText("Payment due today")).toBeVisible(); + }); + + test("shows days remaining mid-month", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T10:00:00") }); + await page.goto("/billing"); + + await expect(page.getByText("16 days until payment")).toBeVisible(); + }); +}); +``` + +## Time Advancement + +### Advance Time Manually + +```typescript +test("session timeout warning", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + await page.goto("/dashboard"); + + // Advance 25 minutes (session timeout at 30 min) + await page.clock.fastForward("25:00"); + + await expect(page.getByText("Session expires in 5 minutes")).toBeVisible(); + + // Advance 5 more minutes + await page.clock.fastForward("05:00"); + + await expect(page.getByText("Session expired")).toBeVisible(); +}); +``` + +### Pause and Resume Time + +```typescript +test("countdown timer", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + await page.goto("/sale"); + + // Initial state + await expect(page.getByText("Sale ends in 2:00:00")).toBeVisible(); + + // Advance 1 hour + await page.clock.fastForward("01:00:00"); + + await expect(page.getByText("Sale ends in 1:00:00")).toBeVisible(); + + // Advance past end + await page.clock.fastForward("01:00:01"); + + await expect(page.getByText("Sale ended")).toBeVisible(); +}); +``` + +### Run Pending Timers + +```typescript +test("debounced search", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + await page.goto("/search"); + + await page.getByLabel("Search").fill("playwright"); + + // Search is debounced by 300ms, won't fire yet + await expect(page.getByTestId("search-results")).toBeHidden(); + + // Fast forward past debounce + await page.clock.fastForward(300); + + // Now search should execute + await expect(page.getByTestId("search-results")).toBeVisible(); +}); +``` + +## Timezone Testing + +### Test Different Timezones + +```typescript +test.describe("timezone display", () => { + test("shows correct time in PST", async ({ browser }) => { + const context = await browser.newContext({ + timezoneId: "America/Los_Angeles", + }); + const page = await context.newPage(); + + await page.clock.install({ time: new Date("2025-01-15T17:00:00Z") }); // 5 PM UTC + + await page.goto("/schedule"); + + // Should show 9 AM PST + await expect(page.getByText("9:00 AM")).toBeVisible(); + + await context.close(); + }); + + test("shows correct time in JST", async ({ browser }) => { + const context = await browser.newContext({ + timezoneId: "Asia/Tokyo", + }); + const page = await context.newPage(); + + await page.clock.install({ time: new Date("2025-01-15T17:00:00Z") }); // 5 PM UTC + + await page.goto("/schedule"); + + // Should show 2 AM next day JST + await expect(page.getByText("2:00 AM")).toBeVisible(); + + await context.close(); + }); +}); +``` + +### Timezone Fixture + +```typescript +// fixtures/timezone.fixture.ts +import { test as base } from "@playwright/test"; + +type TimezoneFixtures = { + pageInTimezone: (timezone: string) => Promise; +}; + +export const test = base.extend({ + pageInTimezone: async ({ browser }, use) => { + const pages: Page[] = []; + + await use(async (timezone) => { + const context = await browser.newContext({ timezoneId: timezone }); + const page = await context.newPage(); + pages.push(page); + return page; + }); + + // Cleanup + for (const page of pages) { + await page.context().close(); + } + }, +}); +``` + +## Timer Mocking + +### Mock setInterval + +```typescript +test("auto-refresh data", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + + let apiCalls = 0; + await page.route("**/api/data", (route) => { + apiCalls++; + route.fulfill({ json: { value: apiCalls } }); + }); + + await page.goto("/live-data"); // Sets up 30s refresh interval + + expect(apiCalls).toBe(1); // Initial load + + // Advance 30 seconds + await page.clock.fastForward("00:30"); + expect(apiCalls).toBe(2); // First refresh + + // Advance another 30 seconds + await page.clock.fastForward("00:30"); + expect(apiCalls).toBe(3); // Second refresh +}); +``` + +### Mock setTimeout Chains + +```typescript +test("notification queue", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + await page.goto("/notifications"); + + // Trigger 3 notifications that show sequentially + await page.getByRole("button", { name: "Show All" }).click(); + + // First notification appears immediately + await expect(page.getByText("Notification 1")).toBeVisible(); + + // Second appears after 2 seconds + await page.clock.fastForward("00:02"); + await expect(page.getByText("Notification 2")).toBeVisible(); + + // Third appears after 2 more seconds + await page.clock.fastForward("00:02"); + await expect(page.getByText("Notification 3")).toBeVisible(); +}); +``` + +### Test Animation Frames + +```typescript +test("animation completes", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); + await page.goto("/animation-demo"); + + await page.getByRole("button", { name: "Animate" }).click(); + + // Animation runs for 500ms + const element = page.getByTestId("animated-box"); + await expect(element).toHaveCSS("opacity", "0"); + + // Fast forward through animation + await page.clock.fastForward(500); + + await expect(element).toHaveCSS("opacity", "1"); +}); +``` + +## Best Practices + +### Always Install Clock Before Navigation + +```typescript +// Good +test("date test", async ({ page }) => { + await page.clock.install({ time: new Date("2025-01-15") }); + await page.goto("/"); // Page loads with mocked time +}); + +// Bad - time already captured by page +test("date test", async ({ page }) => { + await page.goto("/"); + await page.clock.install({ time: new Date("2025-01-15") }); // Too late! +}); +``` + +### Use ISO Strings for Clarity + +```typescript +// Good - explicit timezone +await page.clock.install({ time: new Date("2025-01-15T09:00:00Z") }); + +// Ambiguous - uses local timezone +await page.clock.install({ time: new Date("2025-01-15T09:00:00") }); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ---------------------------------------- | ------------------------------- | -------------------------------------- | +| Installing clock after navigation | Page already captured real time | Install clock before `goto()` | +| Hardcoded relative dates | Tests break over time | Use fixed dates with clock mock | +| Not accounting for timezone | Tests fail in different regions | Use explicit UTC times or set timezone | +| Using `waitForTimeout` with mocked clock | Conflicts with mocked timers | Use `fastForward` instead | + +## Related References + +- **Assertions**: See [assertions-waiting.md](../core/assertions-waiting.md) for time-based assertions +- **Fixtures**: See [fixtures-hooks.md](../core/fixtures-hooks.md) for clock fixtures diff --git a/external/develop/devops/playwright-best-practices/advanced/mobile-testing.md b/external/develop/devops/playwright-best-practices/advanced/mobile-testing.md new file mode 100644 index 0000000..e928bde --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/mobile-testing.md @@ -0,0 +1,409 @@ +# Mobile & Responsive Testing + +## Table of Contents + +1. [Device Emulation](#device-emulation) +2. [Touch Gestures](#touch-gestures) +3. [Viewport Testing](#viewport-testing) +4. [Mobile-Specific UI](#mobile-specific-ui) +5. [Responsive Breakpoints](#responsive-breakpoints) + +## Device Emulation + +### Use Built-in Devices + +```typescript +import { test, devices } from "@playwright/test"; + +// Configure in playwright.config.ts +export default defineConfig({ + projects: [ + { name: "Desktop Chrome", use: { ...devices["Desktop Chrome"] } }, + { name: "Mobile Safari", use: { ...devices["iPhone 14"] } }, + { name: "Mobile Chrome", use: { ...devices["Pixel 7"] } }, + { name: "Tablet", use: { ...devices["iPad Pro 11"] } }, + ], +}); +``` + +### Custom Device Configuration + +```typescript +test.use({ + viewport: { width: 390, height: 844 }, + deviceScaleFactor: 3, + isMobile: true, + hasTouch: true, + userAgent: + "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15", +}); + +test("custom mobile device", async ({ page }) => { + await page.goto("/"); + // Test runs with custom device settings +}); +``` + +### Test Across Multiple Devices + +```typescript +const mobileDevices = ["iPhone 14", "Pixel 7", "Galaxy S21"]; + +for (const deviceName of mobileDevices) { + test(`checkout on ${deviceName}`, async ({ browser }) => { + const device = devices[deviceName]; + const context = await browser.newContext({ ...device }); + const page = await context.newPage(); + + await page.goto("/checkout"); + await expect(page.getByRole("button", { name: "Pay" })).toBeVisible(); + + await context.close(); + }); +} +``` + +## Touch Gestures + +### Tap + +```typescript +test.use({ hasTouch: true }); + +test("tap to interact", async ({ page }) => { + await page.goto("/gallery"); + + // Tap is like click but for touch devices + await page.getByRole("img", { name: "Photo 1" }).tap(); + + await expect(page.getByRole("dialog")).toBeVisible(); +}); +``` + +### Swipe + +```typescript +test("swipe carousel", async ({ page }) => { + await page.goto("/carousel"); + + const carousel = page.getByTestId("carousel"); + const box = await carousel.boundingBox(); + + if (box) { + // Swipe left + await page.touchscreen.tap(box.x + box.width - 50, box.y + box.height / 2); + await page.mouse.move(box.x + 50, box.y + box.height / 2); + + // Or use drag + await carousel.dragTo(carousel, { + sourcePosition: { x: box.width - 50, y: box.height / 2 }, + targetPosition: { x: 50, y: box.height / 2 }, + }); + } + + await expect(page.getByText("Slide 2")).toBeVisible(); +}); +``` + +### Swipe Fixture + +```typescript +// fixtures/touch.fixture.ts +import { test as base, Page } from "@playwright/test"; + +type TouchFixtures = { + swipe: ( + element: Locator, + direction: "left" | "right" | "up" | "down", + ) => Promise; +}; + +export const test = base.extend({ + swipe: async ({ page }, use) => { + await use(async (element, direction) => { + const box = await element.boundingBox(); + if (!box) throw new Error("Element not visible"); + + const centerX = box.x + box.width / 2; + const centerY = box.y + box.height / 2; + const distance = 100; + + const moves = { + left: { + startX: centerX + distance, + endX: centerX - distance, + y: centerY, + }, + right: { + startX: centerX - distance, + endX: centerX + distance, + y: centerY, + }, + up: { + startX: centerX, + endX: centerX, + startY: centerY + distance, + endY: centerY - distance, + }, + down: { + startX: centerX, + endX: centerX, + startY: centerY - distance, + endY: centerY + distance, + }, + }; + + const move = moves[direction]; + await page.touchscreen.tap(move.startX, move.startY ?? move.y); + await page.mouse.move(move.endX, move.endY ?? move.y, { steps: 10 }); + await page.mouse.up(); + }); + }, +}); + +// Usage +test("swipe to delete", async ({ page, swipe }) => { + await page.goto("/inbox"); + + const message = page.getByTestId("message-1"); + await swipe(message, "left"); + + await expect(page.getByRole("button", { name: "Delete" })).toBeVisible(); +}); +``` + +### Long Press + +```typescript +test("long press for context menu", async ({ page }) => { + await page.goto("/files"); + + const file = page.getByText("document.pdf"); + const box = await file.boundingBox(); + + if (box) { + // Touch down + await page.touchscreen.tap(box.x + box.width / 2, box.y + box.height / 2); + + // Hold for 500ms + await page.waitForTimeout(500); + + // Context menu should appear + await expect(page.getByRole("menu")).toBeVisible(); + } +}); +``` + +### Pinch Zoom + +```typescript +test("pinch to zoom image", async ({ page }) => { + await page.goto("/map"); + + // Pinch zoom requires two touch points + // Playwright doesn't have native pinch support, so we simulate via evaluate + await page.evaluate(() => { + const element = document.querySelector("#map"); + if (element) { + // Simulate wheel event as fallback for zoom + element.dispatchEvent( + new WheelEvent("wheel", { + deltaY: -100, // Negative = zoom in + ctrlKey: true, // Ctrl+wheel = pinch on many apps + }), + ); + } + }); + + // Or trigger the app's zoom function directly + await page.evaluate(() => { + (window as any).mapInstance?.setZoom(15); + }); +}); +``` + +## Viewport Testing + +### Test Different Sizes + +```typescript +const viewports = [ + { name: "mobile", width: 375, height: 667 }, + { name: "tablet", width: 768, height: 1024 }, + { name: "desktop", width: 1920, height: 1080 }, +]; + +for (const { name, width, height } of viewports) { + test(`navigation on ${name}`, async ({ page }) => { + await page.setViewportSize({ width, height }); + await page.goto("/"); + + if (width < 768) { + // Mobile: should have hamburger menu + await expect(page.getByRole("button", { name: "Menu" })).toBeVisible(); + } else { + // Desktop: should have visible nav links + await expect(page.getByRole("link", { name: "Products" })).toBeVisible(); + } + }); +} +``` + +### Dynamic Viewport Changes + +```typescript +test("responsive layout change", async ({ page }) => { + await page.setViewportSize({ width: 1200, height: 800 }); + await page.goto("/dashboard"); + + // Desktop: sidebar visible + await expect(page.getByRole("complementary")).toBeVisible(); + + // Resize to mobile + await page.setViewportSize({ width: 375, height: 667 }); + + // Mobile: sidebar hidden, hamburger visible + await expect(page.getByRole("complementary")).toBeHidden(); + await expect(page.getByRole("button", { name: "Menu" })).toBeVisible(); +}); +``` + +## Mobile-Specific UI + +### Hamburger Menu + +```typescript +test("mobile navigation", async ({ page }) => { + await page.setViewportSize({ width: 375, height: 667 }); + await page.goto("/"); + + // Open hamburger menu + await page.getByRole("button", { name: "Menu" }).click(); + + // Navigation drawer should appear + const nav = page.getByRole("navigation"); + await expect(nav).toBeVisible(); + + // Navigate via mobile menu + await nav.getByRole("link", { name: "Products" }).click(); + + await expect(page).toHaveURL("/products"); + // Menu should close after navigation + await expect(nav).toBeHidden(); +}); +``` + +### Bottom Sheet + +```typescript +test("bottom sheet interaction", async ({ page }) => { + await page.setViewportSize({ width: 375, height: 667 }); + await page.goto("/product/123"); + + await page.getByRole("button", { name: "Add to Cart" }).click(); + + // Bottom sheet appears + const sheet = page.getByRole("dialog"); + await expect(sheet).toBeVisible(); + + // Select options + await sheet.getByRole("combobox", { name: "Size" }).selectOption("Large"); + await sheet.getByRole("button", { name: "Confirm" }).click(); + + await expect(page.getByText("Added to cart")).toBeVisible(); +}); +``` + +### Pull to Refresh + +```typescript +test("pull to refresh", async ({ page }) => { + await page.goto("/feed"); + + const feed = page.getByTestId("feed"); + const initialFirstItem = await feed.locator("> *").first().textContent(); + + // Simulate pull down + const box = await feed.boundingBox(); + if (box) { + await page.touchscreen.tap(box.x + box.width / 2, box.y + 50); + await page.mouse.move(box.x + box.width / 2, box.y + 200, { steps: 20 }); + await page.mouse.up(); + } + + // Wait for refresh + await expect(page.getByTestId("loading")).toBeVisible(); + await expect(page.getByTestId("loading")).toBeHidden(); + + // Content should be updated (in a real app) +}); +``` + +## Responsive Breakpoints + +### Test All Breakpoints + +```typescript +const breakpoints = { + xs: 320, + sm: 640, + md: 768, + lg: 1024, + xl: 1280, + "2xl": 1536, +}; + +test.describe("responsive header", () => { + for (const [name, width] of Object.entries(breakpoints)) { + test(`header at ${name} (${width}px)`, async ({ page }) => { + await page.setViewportSize({ width, height: 800 }); + await page.goto("/"); + + if (width < 768) { + await expect(page.getByTestId("mobile-menu-button")).toBeVisible(); + await expect(page.getByTestId("desktop-nav")).toBeHidden(); + } else { + await expect(page.getByTestId("mobile-menu-button")).toBeHidden(); + await expect(page.getByTestId("desktop-nav")).toBeVisible(); + } + }); + } +}); +``` + +### Visual Regression at Breakpoints + +```typescript +test.describe("visual regression", () => { + const sizes = [ + { width: 375, height: 667, name: "mobile" }, + { width: 768, height: 1024, name: "tablet" }, + { width: 1440, height: 900, name: "desktop" }, + ]; + + for (const { width, height, name } of sizes) { + test(`homepage at ${name}`, async ({ page }) => { + await page.setViewportSize({ width, height }); + await page.goto("/"); + + await expect(page).toHaveScreenshot(`homepage-${name}.png`); + }); + } +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| --------------------------- | ------------------------- | -------------------------------- | +| Only testing one viewport | Misses responsive bugs | Test multiple breakpoints | +| Ignoring touch events | Features broken on mobile | Test tap, swipe, long press | +| Hardcoded viewport in tests | Can't test multiple sizes | Use `page.setViewportSize()` | +| Not testing orientation | Landscape bugs missed | Test both portrait and landscape | + +## Related References + +- **Visual Testing**: See [test-suite-structure.md](../core/test-suite-structure.md) for screenshot testing +- **Locators**: See [locators.md](../core/locators.md) for mobile-friendly selectors +- **Browser APIs**: See [browser-apis.md](../browser-apis/browser-apis.md) for permissions (camera, geolocation, notifications) +- **Canvas/Touch**: See [canvas-webgl.md](../testing-patterns/canvas-webgl.md) for touch gestures on canvas elements diff --git a/external/develop/devops/playwright-best-practices/advanced/multi-context.md b/external/develop/devops/playwright-best-practices/advanced/multi-context.md new file mode 100644 index 0000000..ed1cf8a --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/multi-context.md @@ -0,0 +1,288 @@ +# Multi-Tab, Window & Popup Testing + +This file covers **single-user scenarios** with multiple browser tabs, windows, and popups. For **multi-user collaboration testing** (multiple users interacting simultaneously), see [multi-user.md](multi-user.md). + +## Table of Contents + +1. [Popup Handling](#popup-handling) +2. [New Tab Navigation](#new-tab-navigation) +3. [OAuth Flows](#oauth-flows) +4. [Multiple Windows](#multiple-windows) +5. [Tab Coordination](#tab-coordination) + +## Popup Handling + +### Basic Popup + +```typescript +test("handle popup window", async ({ page }) => { + await page.goto("/"); + + // Start waiting for popup before triggering it + const popupPromise = page.waitForEvent("popup"); + await page.getByRole("button", { name: "Open Support Chat" }).click(); + const popup = await popupPromise; + + // Wait for popup to load + await popup.waitForLoadState(); + + // Interact with popup + await popup.getByLabel("Message").fill("Need help"); + await popup.getByRole("button", { name: "Send" }).click(); + + await expect(popup.getByText("Message sent")).toBeVisible(); + + // Close popup + await popup.close(); +}); +``` + +### Popup with Authentication + +```typescript +test("popup login flow", async ({ page }) => { + await page.goto("/dashboard"); + + const popupPromise = page.waitForEvent("popup"); + await page.getByRole("button", { name: "Connect Account" }).click(); + const popup = await popupPromise; + + await popup.waitForLoadState(); + + // Complete login in popup + await popup.getByLabel("Email").fill("user@example.com"); + await popup.getByLabel("Password").fill("password123"); + await popup.getByRole("button", { name: "Log In" }).click(); + + // Popup should close automatically after auth + await popup.waitForEvent("close"); + + // Main page should reflect connected state + await expect(page.getByText("Account connected")).toBeVisible(); +}); +``` + +### Handle Blocked Popups + +```typescript +test("handle popup blocker", async ({ page }) => { + await page.goto("/share"); + + // Listen for console messages about blocked popup + page.on("console", (msg) => { + if (msg.text().includes("popup blocked")) { + console.log("Popup was blocked"); + } + }); + + const popupPromise = page.waitForEvent("popup").catch(() => null); + await page.getByRole("button", { name: "Share to Twitter" }).click(); + const popup = await popupPromise; + + if (!popup) { + // Popup blocked - app should show fallback + await expect(page.getByText("Copy share link instead")).toBeVisible(); + } +}); +``` + +## New Tab Navigation + +### Link Opens in New Tab + +```typescript +test("external link opens in new tab", async ({ page, context }) => { + await page.goto("/resources"); + + // Wait for new page in context + const pagePromise = context.waitForEvent("page"); + await page.getByRole("link", { name: "Documentation" }).click(); + const newPage = await pagePromise; + + await newPage.waitForLoadState(); + + expect(newPage.url()).toContain("docs.example.com"); + await expect(newPage.getByRole("heading", { level: 1 })).toBeVisible(); + + // Original page still there + expect(page.url()).toContain("/resources"); + + await newPage.close(); +}); +``` + +### Intercept New Tab + +```typescript +test("prevent new tab for testing", async ({ page }) => { + await page.goto("/links"); + + // Remove target="_blank" to keep navigation in same tab + await page.evaluate(() => { + document.querySelectorAll('a[target="_blank"]').forEach((a) => { + a.removeAttribute("target"); + }); + }); + + // Now link opens in same tab + await page.getByRole("link", { name: "External Site" }).click(); + + // Can test the destination page + await expect(page).toHaveURL(/external-site\.com/); +}); +``` + +## OAuth Flows + +### Google OAuth Popup + +```typescript +test("Google OAuth login", async ({ page }) => { + await page.goto("/login"); + + const popupPromise = page.waitForEvent("popup"); + await page.getByRole("button", { name: "Sign in with Google" }).click(); + const popup = await popupPromise; + + await popup.waitForLoadState(); + + // Handle Google's OAuth flow + await popup.getByLabel("Email or phone").fill("test@gmail.com"); + await popup.getByRole("button", { name: "Next" }).click(); + + await popup.getByLabel("Enter your password").fill("password"); + await popup.getByRole("button", { name: "Next" }).click(); + + // Wait for redirect back and popup close + await popup.waitForEvent("close"); + + // Verify logged in on main page + await expect(page.getByText("Welcome, Test User")).toBeVisible(); +}); +``` + +### Mock OAuth (Recommended) + +```typescript +test("mock OAuth flow", async ({ page, context }) => { + // Mock the OAuth callback instead of real flow + await page.route("**/auth/callback**", async (route) => { + // Simulate successful OAuth + const url = new URL(route.request().url()); + url.searchParams.set("code", "mock-auth-code"); + await route.fulfill({ + status: 302, + headers: { Location: "/dashboard" }, + }); + }); + + // Mock token exchange + await page.route("**/api/auth/token", (route) => + route.fulfill({ + json: { + access_token: "mock-token", + user: { name: "Test User", email: "test@example.com" }, + }, + }), + ); + + await page.goto("/login"); + await page.getByRole("button", { name: "Sign in with Google" }).click(); + + // Should redirect to dashboard without actual OAuth + await expect(page).toHaveURL("/dashboard"); + await expect(page.getByText("Welcome, Test User")).toBeVisible(); +}); +``` + +### OAuth Fixture + +> **For comprehensive OAuth mocking patterns** (fixtures, multiple providers, SAML SSO), see [third-party.md](third-party.md#oauthsso-mocking). This section focuses on popup window handling mechanics for OAuth flows. + +## Multiple Windows + +### Test Across Multiple Windows + +```typescript +test("sync between windows", async ({ context }) => { + // Open two pages + const page1 = await context.newPage(); + const page2 = await context.newPage(); + + await page1.goto("/dashboard"); + await page2.goto("/dashboard"); + + // Make change in first window + await page1.getByRole("button", { name: "Add Item" }).click(); + await page1.getByLabel("Name").fill("New Item"); + await page1.getByRole("button", { name: "Save" }).click(); + + // Should sync to second window (if app supports real-time sync) + await expect(page2.getByText("New Item")).toBeVisible({ timeout: 10000 }); +}); +``` + +### Different Users in Different Windows + +> **For multi-user collaboration patterns** (admin/user interactions, real-time collaboration, role-based testing, concurrent actions), see [multi-user.md](multi-user.md). This file focuses on single-user scenarios with multiple tabs/windows/popups. + +## Tab Coordination + +### Switch Between Tabs + +```typescript +test("manage multiple tabs", async ({ context }) => { + const page1 = await context.newPage(); + await page1.goto("/editor"); + + const page2 = await context.newPage(); + await page2.goto("/preview"); + + // Edit in first tab + await page1.bringToFront(); + await page1.getByLabel("Content").fill("Hello World"); + + // Check preview in second tab + await page2.bringToFront(); + await page2.reload(); // If preview needs refresh + await expect(page2.getByText("Hello World")).toBeVisible(); +}); +``` + +### Close All Tabs Except One + +```typescript +test("cleanup tabs after test", async ({ context }) => { + const mainPage = await context.newPage(); + await mainPage.goto("/"); + + // Open several popups during test + for (let i = 0; i < 3; i++) { + const popup = await context.newPage(); + await popup.goto(`/popup/${i}`); + } + + // Close all except main page + for (const page of context.pages()) { + if (page !== mainPage) { + await page.close(); + } + } + + expect(context.pages()).toHaveLength(1); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ----------------------- | ------------------------------ | ------------------------------------------ | +| Not waiting for popup | Race condition | Use `waitForEvent("popup")` before trigger | +| Testing real OAuth | Slow, flaky, needs credentials | Mock OAuth endpoints | +| Assuming popup opens | May be blocked | Handle both open and blocked cases | +| Not closing extra pages | Resource leak | Close pages in cleanup | + +## Related References + +- **Authentication**: See [fixtures-hooks.md](../core/fixtures-hooks.md) for auth patterns +- **Network**: See [network-advanced.md](network-advanced.md) for mocking OAuth diff --git a/external/develop/devops/playwright-best-practices/advanced/multi-user.md b/external/develop/devops/playwright-best-practices/advanced/multi-user.md new file mode 100644 index 0000000..301e55c --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/multi-user.md @@ -0,0 +1,393 @@ +# Multi-User & Collaboration Testing + +## Table of Contents + +1. [Multiple Browser Contexts](#multiple-browser-contexts) +2. [Real-Time Collaboration](#real-time-collaboration) +3. [Role-Based Testing](#role-based-testing) +4. [Concurrent Actions](#concurrent-actions) +5. [Chat & Messaging](#chat--messaging) + +## Multiple Browser Contexts + +### Two Users in Same Test + +```typescript +test("two users see each other's changes", async ({ browser }) => { + // Create two isolated contexts (like two browsers) + const userAContext = await browser.newContext(); + const userBContext = await browser.newContext(); + + const userAPage = await userAContext.newPage(); + const userBPage = await userBContext.newPage(); + + // Both users go to the same document + await userAPage.goto("/doc/shared-123"); + await userBPage.goto("/doc/shared-123"); + + // User A types + await userAPage.getByLabel("Content").fill("Hello from User A"); + + // User B should see the change + await expect(userBPage.getByText("Hello from User A")).toBeVisible(); + + // Cleanup + await userAContext.close(); + await userBContext.close(); +}); +``` + +### Multiple Users with Auth States + +```typescript +test("admin and user interaction", async ({ browser }) => { + // Load different auth states + const adminContext = await browser.newContext({ + storageState: ".auth/admin.json", + }); + const userContext = await browser.newContext({ + storageState: ".auth/user.json", + }); + + const adminPage = await adminContext.newPage(); + const userPage = await userContext.newPage(); + + // User submits request + await userPage.goto("/support"); + await userPage.getByLabel("Message").fill("Need help!"); + await userPage.getByRole("button", { name: "Submit" }).click(); + + // Admin sees and responds + await adminPage.goto("/admin/tickets"); + await expect(adminPage.getByText("Need help!")).toBeVisible(); + await adminPage.getByRole("button", { name: "Reply" }).click(); + await adminPage.getByLabel("Response").fill("How can I help?"); + await adminPage.getByRole("button", { name: "Send" }).click(); + + // User sees response + await expect(userPage.getByText("How can I help?")).toBeVisible(); + + await adminContext.close(); + await userContext.close(); +}); +``` + +### Multi-User Fixture + +```typescript +// fixtures/multi-user.fixture.ts +import { test as base, Browser, BrowserContext, Page } from "@playwright/test"; + +type UserSession = { + context: BrowserContext; + page: Page; +}; + +type MultiUserFixtures = { + createUser: (authState?: string) => Promise; +}; + +export const test = base.extend({ + createUser: async ({ browser }, use) => { + const sessions: UserSession[] = []; + + await use(async (authState) => { + const context = await browser.newContext({ + storageState: authState, + }); + const page = await context.newPage(); + sessions.push({ context, page }); + return { context, page }; + }); + + // Cleanup all sessions + for (const session of sessions) { + await session.context.close(); + } + }, +}); + +// Usage +test("3 users collaborate", async ({ createUser }) => { + const alice = await createUser(".auth/alice.json"); + const bob = await createUser(".auth/bob.json"); + const charlie = await createUser(".auth/charlie.json"); + + // All navigate to same room + await alice.page.goto("/room/123"); + await bob.page.goto("/room/123"); + await charlie.page.goto("/room/123"); + + // Test interactions... +}); +``` + +## Real-Time Collaboration + +### Collaborative Document + +```typescript +test("real-time collaborative editing", async ({ browser }) => { + const user1 = await browser.newContext(); + const user2 = await browser.newContext(); + + const page1 = await user1.newPage(); + const page2 = await user2.newPage(); + + await page1.goto("/docs/shared"); + await page2.goto("/docs/shared"); + + // User 1 types at the beginning + const editor1 = page1.getByRole("textbox"); + await editor1.click(); + await editor1.press("Home"); + await editor1.type("User 1: "); + + // User 2 types at the end + const editor2 = page2.getByRole("textbox"); + await editor2.click(); + await editor2.press("End"); + await editor2.type(" - User 2"); + + // Both should see combined result + await expect(page1.getByRole("textbox")).toContainText("User 1:"); + await expect(page1.getByRole("textbox")).toContainText("- User 2"); + await expect(page2.getByRole("textbox")).toContainText("User 1:"); + await expect(page2.getByRole("textbox")).toContainText("- User 2"); + + await user1.close(); + await user2.close(); +}); +``` + +### Cursor Presence + +```typescript +test("shows other user cursors", async ({ browser }) => { + const ctx1 = await browser.newContext(); + const ctx2 = await browser.newContext(); + + const page1 = await ctx1.newPage(); + const page2 = await ctx2.newPage(); + + // Mock to identify users + await page1.route("**/api/me", (route) => + route.fulfill({ json: { id: "user-1", name: "Alice" } }), + ); + await page2.route("**/api/me", (route) => + route.fulfill({ json: { id: "user-2", name: "Bob" } }), + ); + + await page1.goto("/whiteboard/123"); + await page2.goto("/whiteboard/123"); + + // Move cursor on page1 + await page1.mouse.move(200, 200); + + // Page2 should see Alice's cursor + await expect(page2.getByTestId("cursor-user-1")).toBeVisible(); + await expect(page2.getByText("Alice")).toBeVisible(); + + await ctx1.close(); + await ctx2.close(); +}); +``` + +## Role-Based Testing + +### Test RBAC + +```typescript +const roles = [ + { role: "admin", canDelete: true, canEdit: true, canView: true }, + { role: "editor", canDelete: false, canEdit: true, canView: true }, + { role: "viewer", canDelete: false, canEdit: false, canView: true }, +]; + +for (const { role, canDelete, canEdit, canView } of roles) { + test(`${role} permissions`, async ({ browser }) => { + const context = await browser.newContext({ + storageState: `.auth/${role}.json`, + }); + const page = await context.newPage(); + + await page.goto("/document/123"); + + // Check view permission + if (canView) { + await expect(page.getByTestId("content")).toBeVisible(); + } else { + await expect(page.getByText("Access denied")).toBeVisible(); + } + + // Check edit permission + const editButton = page.getByRole("button", { name: "Edit" }); + if (canEdit) { + await expect(editButton).toBeEnabled(); + } else { + await expect(editButton).toBeDisabled(); + } + + // Check delete permission + const deleteButton = page.getByRole("button", { name: "Delete" }); + if (canDelete) { + await expect(deleteButton).toBeVisible(); + } else { + await expect(deleteButton).toBeHidden(); + } + + await context.close(); + }); +} +``` + +### Permission Escalation Test + +```typescript +test("cannot access admin routes as user", async ({ browser }) => { + const userContext = await browser.newContext({ + storageState: ".auth/user.json", + }); + const page = await userContext.newPage(); + + // Try to access admin page directly + await page.goto("/admin/users"); + + // Should redirect or show error + await expect(page).not.toHaveURL("/admin/users"); + await expect(page.getByText("Access denied")).toBeVisible(); + + await userContext.close(); +}); +``` + +## Concurrent Actions + +### Race Condition Testing + +```typescript +test("handles concurrent edits", async ({ browser }) => { + const ctx1 = await browser.newContext(); + const ctx2 = await browser.newContext(); + + const page1 = await ctx1.newPage(); + const page2 = await ctx2.newPage(); + + await page1.goto("/item/123"); + await page2.goto("/item/123"); + + // Both click edit at the same time + await Promise.all([ + page1.getByRole("button", { name: "Edit" }).click(), + page2.getByRole("button", { name: "Edit" }).click(), + ]); + + // Both try to save different values + await page1.getByLabel("Name").fill("Value from User 1"); + await page2.getByLabel("Name").fill("Value from User 2"); + + await Promise.all([ + page1.getByRole("button", { name: "Save" }).click(), + page2.getByRole("button", { name: "Save" }).click(), + ]); + + // One should succeed, one should get conflict error + const page1HasConflict = await page1.getByText("Conflict").isVisible(); + const page2HasConflict = await page2.getByText("Conflict").isVisible(); + + // Exactly one should have conflict + expect(page1HasConflict || page2HasConflict).toBe(true); + expect(page1HasConflict && page2HasConflict).toBe(false); + + await ctx1.close(); + await ctx2.close(); +}); +``` + +### Optimistic Locking Test + +```typescript +test("optimistic locking prevents overwrites", async ({ browser }) => { + const ctx1 = await browser.newContext(); + const ctx2 = await browser.newContext(); + + const page1 = await ctx1.newPage(); + const page2 = await ctx2.newPage(); + + // Both load the same version + await page1.goto("/record/123"); + await page2.goto("/record/123"); + + // User 1 edits and saves first + await page1.getByRole("button", { name: "Edit" }).click(); + await page1.getByLabel("Value").fill("Updated by User 1"); + await page1.getByRole("button", { name: "Save" }).click(); + await expect(page1.getByText("Saved")).toBeVisible(); + + // User 2 tries to save with stale version + await page2.getByRole("button", { name: "Edit" }).click(); + await page2.getByLabel("Value").fill("Updated by User 2"); + await page2.getByRole("button", { name: "Save" }).click(); + + // Should fail with version conflict + await expect(page2.getByText("Someone else modified this")).toBeVisible(); + await expect(page2.getByRole("button", { name: "Reload" })).toBeVisible(); + + await ctx1.close(); + await ctx2.close(); +}); +``` + +## Chat & Messaging + +### Real-Time Chat + +```typescript +test("chat messages sync between users", async ({ browser }) => { + const aliceCtx = await browser.newContext(); + const bobCtx = await browser.newContext(); + + const alicePage = await aliceCtx.newPage(); + const bobPage = await bobCtx.newPage(); + + // Setup user identities + await alicePage.route("**/api/me", (r) => + r.fulfill({ json: { name: "Alice" } }), + ); + await bobPage.route("**/api/me", (r) => r.fulfill({ json: { name: "Bob" } })); + + await alicePage.goto("/chat/room-1"); + await bobPage.goto("/chat/room-1"); + + // Alice sends message + await alicePage.getByLabel("Message").fill("Hi Bob!"); + await alicePage.getByRole("button", { name: "Send" }).click(); + + // Bob sees it + await expect(bobPage.getByText("Alice: Hi Bob!")).toBeVisible(); + + // Bob replies + await bobPage.getByLabel("Message").fill("Hey Alice!"); + await bobPage.getByRole("button", { name: "Send" }).click(); + + // Alice sees it + await expect(alicePage.getByText("Bob: Hey Alice!")).toBeVisible(); + + await aliceCtx.close(); + await bobCtx.close(); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ----------------------------- | ----------------------------- | ---------------------------- | +| Sharing context between users | State leaks, not isolated | Create separate contexts | +| Not closing contexts | Memory leak, browser overload | Always close in cleanup | +| Hardcoded timing for sync | Flaky tests | Use `expect().toBeVisible()` | +| Testing only single user | Misses collaboration bugs | Test multi-user scenarios | + +## Related References + +- **Authentication**: See [fixtures-hooks.md](../core/fixtures-hooks.md) for auth setup +- **WebSockets**: See [websockets.md](../browser-apis/websockets.md) for real-time mocking diff --git a/external/develop/devops/playwright-best-practices/advanced/network-advanced.md b/external/develop/devops/playwright-best-practices/advanced/network-advanced.md new file mode 100644 index 0000000..fa017fe --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/network-advanced.md @@ -0,0 +1,452 @@ +# Advanced Network Interception + +## Table of Contents + +1. [Request Modification](#request-modification) +2. [GraphQL Mocking](#graphql-mocking) +3. [HAR Recording & Playback](#har-recording--playback) +4. [Conditional Mocking](#conditional-mocking) +5. [Network Throttling](#network-throttling) + +## Request Modification + +### Modify Request Headers + +```typescript +test("add auth header to requests", async ({ page }) => { + await page.route("**/api/**", (route) => { + const headers = { + ...route.request().headers(), + Authorization: "Bearer test-token", + "X-Test-Header": "test-value", + }; + route.continue({ headers }); + }); + + await page.goto("/dashboard"); +}); +``` + +### Modify Request Body + +```typescript +test("modify POST body", async ({ page }) => { + await page.route("**/api/orders", async (route) => { + if (route.request().method() === "POST") { + const postData = route.request().postDataJSON(); + + // Add test metadata + const modifiedData = { + ...postData, + testMode: true, + testTimestamp: Date.now(), + }; + + await route.continue({ + postData: JSON.stringify(modifiedData), + }); + } else { + await route.continue(); + } + }); + + await page.goto("/checkout"); + await page.getByRole("button", { name: "Place Order" }).click(); +}); +``` + +### Transform Response + +```typescript +test("modify API response", async ({ page }) => { + await page.route("**/api/products", async (route) => { + // Fetch real response + const response = await route.fetch(); + const json = await response.json(); + + // Modify response + const modified = json.map((product: any) => ({ + ...product, + price: product.price * 0.9, // 10% discount + testMode: true, + })); + + await route.fulfill({ + response, + json: modified, + }); + }); + + await page.goto("/products"); +}); +``` + +## GraphQL Mocking + +### Mock by Operation Name + +```typescript +test("mock GraphQL query", async ({ page }) => { + await page.route("**/graphql", async (route) => { + const postData = route.request().postDataJSON(); + + if (postData.operationName === "GetUser") { + return route.fulfill({ + json: { + data: { + user: { + id: "1", + name: "Test User", + email: "test@example.com", + }, + }, + }, + }); + } + + if (postData.operationName === "GetProducts") { + return route.fulfill({ + json: { + data: { + products: [ + { id: "1", name: "Product A", price: 29.99 }, + { id: "2", name: "Product B", price: 49.99 }, + ], + }, + }, + }); + } + + // Pass through unmocked operations + return route.continue(); + }); + + await page.goto("/dashboard"); +}); +``` + +### GraphQL Mock Fixture + +```typescript +// fixtures/graphql.fixture.ts +type GraphQLMock = { + operation: string; + variables?: Record; + response: { data?: any; errors?: any[] }; +}; + +type GraphQLFixtures = { + mockGraphQL: (mocks: GraphQLMock[]) => Promise; +}; + +export const test = base.extend({ + mockGraphQL: async ({ page }, use) => { + await use(async (mocks) => { + await page.route("**/graphql", async (route) => { + const postData = route.request().postDataJSON(); + + const mock = mocks.find((m) => { + if (m.operation !== postData.operationName) return false; + + // Optionally match variables + if (m.variables) { + return ( + JSON.stringify(m.variables) === JSON.stringify(postData.variables) + ); + } + return true; + }); + + if (mock) { + return route.fulfill({ json: mock.response }); + } + + return route.continue(); + }); + }); + }, +}); + +// Usage +test("dashboard with mocked GraphQL", async ({ page, mockGraphQL }) => { + await mockGraphQL([ + { + operation: "GetDashboardStats", + response: { + data: { stats: { users: 100, revenue: 50000 } }, + }, + }, + { + operation: "GetUser", + variables: { id: "1" }, + response: { + data: { user: { id: "1", name: "John" } }, + }, + }, + ]); + + await page.goto("/dashboard"); + await expect(page.getByText("100 users")).toBeVisible(); +}); +``` + +### Mock GraphQL Mutations + +```typescript +test("mock GraphQL mutation", async ({ page }) => { + await page.route("**/graphql", async (route) => { + const postData = route.request().postDataJSON(); + + if (postData.operationName === "CreateOrder") { + const { input } = postData.variables; + + return route.fulfill({ + json: { + data: { + createOrder: { + id: "order-123", + status: "PENDING", + items: input.items, + total: input.items.reduce( + (sum: number, item: any) => sum + item.price * item.quantity, + 0, + ), + }, + }, + }, + }); + } + + return route.continue(); + }); + + await page.goto("/checkout"); + await page.getByRole("button", { name: "Place Order" }).click(); + + await expect(page.getByText("Order #order-123")).toBeVisible(); +}); +``` + +## HAR Recording & Playback + +### Record HAR File + +```typescript +// Record network traffic +test("record HAR", async ({ page, context }) => { + // Start recording + await context.routeFromHAR("./recordings/checkout.har", { + update: true, // Create/update HAR file + url: "**/api/**", + }); + + await page.goto("/checkout"); + await page.getByRole("button", { name: "Place Order" }).click(); + + // HAR file is saved automatically +}); +``` + +### Playback HAR File + +```typescript +// Use recorded HAR for offline testing +test("playback HAR", async ({ page, context }) => { + await context.routeFromHAR("./recordings/checkout.har", { + url: "**/api/**", + update: false, // Don't update, just playback + }); + + await page.goto("/checkout"); + + // All API calls served from HAR file + await expect(page.getByText("Order confirmed")).toBeVisible(); +}); +``` + +### HAR with Fallback + +```typescript +test("HAR with live fallback", async ({ page, context }) => { + await context.routeFromHAR("./recordings/api.har", { + url: "**/api/**", + update: false, + notFound: "fallback", // Use real network if not in HAR + }); + + await page.goto("/dashboard"); +}); +``` + +## Conditional Mocking + +### Mock Based on Request Body + +```typescript +test("conditional mock by body", async ({ page }) => { + await page.route("**/api/search", async (route) => { + const body = route.request().postDataJSON(); + + if (body.query === "error") { + return route.fulfill({ + status: 500, + json: { error: "Search failed" }, + }); + } + + if (body.query === "empty") { + return route.fulfill({ + json: { results: [] }, + }); + } + + // Default response + return route.fulfill({ + json: { + results: [{ id: 1, title: `Result for: ${body.query}` }], + }, + }); + }); + + await page.goto("/search"); + + // Test different scenarios + await page.getByLabel("Search").fill("error"); + await page.getByLabel("Search").press("Enter"); + await expect(page.getByText("Search failed")).toBeVisible(); +}); +``` + +### Mock Nth Request + +```typescript +test("different response on retry", async ({ page }) => { + let callCount = 0; + + await page.route("**/api/status", (route) => { + callCount++; + + if (callCount < 3) { + return route.fulfill({ + status: 503, + json: { error: "Service unavailable" }, + }); + } + + // Succeed on 3rd attempt + return route.fulfill({ + json: { status: "ok" }, + }); + }); + + await page.goto("/dashboard"); + + // App should retry and eventually succeed + await expect(page.getByText("Connected")).toBeVisible(); +}); +``` + +### Mock with Delay + +```typescript +test("slow network simulation", async ({ page }) => { + await page.route("**/api/data", async (route) => { + // Simulate 2 second delay + await new Promise((resolve) => setTimeout(resolve, 2000)); + + return route.fulfill({ + json: { data: "loaded" }, + }); + }); + + await page.goto("/dashboard"); + + // Loading state should appear + await expect(page.getByText("Loading...")).toBeVisible(); + + // Then data appears + await expect(page.getByText("loaded")).toBeVisible(); +}); +``` + +## Network Throttling + +### Slow 3G Simulation + +```typescript +test("slow network experience", async ({ page, context }) => { + // Create CDP session for network throttling + const client = await context.newCDPSession(page); + + await client.send("Network.emulateNetworkConditions", { + offline: false, + downloadThroughput: (500 * 1024) / 8, // 500 Kbps + uploadThroughput: (500 * 1024) / 8, + latency: 400, // 400ms + }); + + await page.goto("/"); + + // Test loading states appear + await expect(page.getByTestId("skeleton-loader")).toBeVisible(); +}); +``` + +### Offline Mode + +Use `context.setOffline(true/false)` to simulate network connectivity changes. + +> **For comprehensive offline testing patterns:** +> +> - **Network failure simulation** (error recovery, graceful degradation): See [error-testing.md](error-testing.md#offline-testing) +> - **Offline-first/PWA testing** (service workers, caching, background sync): See [service-workers.md](service-workers.md#offline-testing) + +### Network Throttling Fixture + +```typescript +// fixtures/network.fixture.ts +type NetworkCondition = "slow3g" | "fast3g" | "offline"; + +const conditions = { + slow3g: { downloadThroughput: 50000, uploadThroughput: 50000, latency: 2000 }, + fast3g: { downloadThroughput: 180000, uploadThroughput: 75000, latency: 150 }, +}; + +type NetworkFixtures = { + setNetworkCondition: (condition: NetworkCondition) => Promise; +}; + +export const test = base.extend({ + setNetworkCondition: async ({ page, context }, use) => { + const client = await context.newCDPSession(page); + + await use(async (condition) => { + if (condition === "offline") { + await context.setOffline(true); + } else { + await client.send("Network.emulateNetworkConditions", { + offline: false, + ...conditions[condition], + }); + } + }); + + // Reset + await context.setOffline(false); + }, +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ------------------------ | ------------------------------ | -------------------------------- | +| Mocking all requests | Tests don't reflect reality | Mock only what's necessary | +| No cleanup of routes | Routes persist across tests | Use fixtures with cleanup | +| Ignoring request method | Mock applies to wrong requests | Check `route.request().method()` | +| Hardcoded mock responses | Brittle, hard to maintain | Use factories for mock data | + +## Related References + +- **Basic Mocking**: See [test-suite-structure.md](../core/test-suite-structure.md) for simple mocking +- **WebSockets**: See [websockets.md](../browser-apis/websockets.md) for real-time mocking diff --git a/external/develop/devops/playwright-best-practices/advanced/third-party.md b/external/develop/devops/playwright-best-practices/advanced/third-party.md new file mode 100644 index 0000000..acf8ab8 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/advanced/third-party.md @@ -0,0 +1,464 @@ +# Third-Party Service Mocking + +## Table of Contents + +1. [OAuth/SSO Mocking](#oauthsso-mocking) +2. [Payment Gateway Mocking](#payment-gateway-mocking) +3. [Email Verification](#email-verification) +4. [SMS Verification](#sms-verification) +5. [Analytics & Tracking](#analytics--tracking) + +## OAuth/SSO Mocking + +### Mock Google OAuth + +```typescript +test("Google OAuth login", async ({ page }) => { + // Mock the OAuth callback + await page.route("**/auth/google/callback**", (route) => { + const url = new URL(route.request().url()); + // Simulate successful OAuth by redirecting with token + route.fulfill({ + status: 302, + headers: { + Location: "/dashboard?token=mock-jwt-token", + }, + }); + }); + + // Mock the token verification endpoint + await page.route("**/api/auth/verify", (route) => + route.fulfill({ + json: { + valid: true, + user: { + id: "123", + email: "test@gmail.com", + name: "Test User", + }, + }, + }), + ); + + await page.goto("/login"); + await page.getByRole("button", { name: "Sign in with Google" }).click(); + + await expect(page.getByText("Welcome, Test User")).toBeVisible(); +}); +``` + +### OAuth Fixture + +```typescript +// fixtures/oauth.fixture.ts +type OAuthProvider = "google" | "github" | "microsoft"; + +type OAuthUser = { + id: string; + email: string; + name: string; + avatar?: string; +}; + +type OAuthFixtures = { + mockOAuth: (provider: OAuthProvider, user: OAuthUser) => Promise; +}; + +export const test = base.extend({ + mockOAuth: async ({ page }, use) => { + await use(async (provider, user) => { + // Mock callback redirect + await page.route(`**/auth/${provider}/callback**`, (route) => + route.fulfill({ + status: 302, + headers: { Location: `/auth/success?provider=${provider}` }, + }), + ); + + // Mock session/user endpoint + await page.route("**/api/auth/session", (route) => + route.fulfill({ + json: { user, provider, authenticated: true }, + }), + ); + + // Mock user info endpoint + await page.route("**/api/me", (route) => route.fulfill({ json: user })); + }); + }, +}); + +// Usage +test("login with GitHub", async ({ page, mockOAuth }) => { + await mockOAuth("github", { + id: "gh-123", + email: "dev@github.com", + name: "GitHub User", + }); + + await page.goto("/login"); + await page.getByRole("button", { name: "Sign in with GitHub" }).click(); + + await expect(page.getByText("Welcome, GitHub User")).toBeVisible(); +}); +``` + +### Mock SAML SSO + +```typescript +test("SAML SSO login", async ({ page }) => { + // Mock SAML assertion consumer service + await page.route("**/saml/acs", async (route) => { + route.fulfill({ + status: 302, + headers: { + Location: "/dashboard", + "Set-Cookie": "session=mock-saml-session; Path=/; HttpOnly", + }, + }); + }); + + // Mock session validation + await page.route("**/api/session", (route) => + route.fulfill({ + json: { + user: { email: "user@company.com", name: "SSO User" }, + provider: "saml", + }, + }), + ); + + await page.goto("/login"); + await page.getByRole("button", { name: "SSO Login" }).click(); + + await expect(page).toHaveURL("/dashboard"); +}); +``` + +## Payment Gateway Mocking + +### Mock Stripe + +```typescript +test("Stripe checkout", async ({ page }) => { + // Mock Stripe.js + await page.addInitScript(() => { + (window as any).Stripe = () => ({ + elements: () => ({ + create: () => ({ + mount: () => {}, + on: () => {}, + destroy: () => {}, + }), + }), + confirmCardPayment: async () => ({ + paymentIntent: { status: "succeeded", id: "pi_mock_123" }, + }), + createPaymentMethod: async () => ({ + paymentMethod: { id: "pm_mock_123" }, + }), + }); + }); + + // Mock backend payment endpoint + await page.route("**/api/create-payment-intent", (route) => + route.fulfill({ + json: { clientSecret: "pi_mock_123_secret_mock" }, + }), + ); + + await page.route("**/api/confirm-payment", (route) => + route.fulfill({ + json: { success: true, orderId: "order-123" }, + }), + ); + + await page.goto("/checkout"); + await page.getByRole("button", { name: "Pay $99.99" }).click(); + + await expect(page.getByText("Payment successful")).toBeVisible(); +}); +``` + +### Mock PayPal + +```typescript +test("PayPal checkout", async ({ page }) => { + // Mock PayPal SDK + await page.addInitScript(() => { + (window as any).paypal = { + Buttons: () => ({ + render: () => Promise.resolve(), + isEligible: () => true, + }), + FUNDING: { PAYPAL: "paypal", CARD: "card" }, + }; + }); + + // Mock PayPal order creation + await page.route("**/api/paypal/create-order", (route) => + route.fulfill({ + json: { orderId: "PAYPAL-ORDER-123" }, + }), + ); + + // Mock PayPal capture + await page.route("**/api/paypal/capture", (route) => + route.fulfill({ + json: { success: true, transactionId: "TXN-123" }, + }), + ); + + await page.goto("/checkout"); + + // Simulate PayPal approval callback + await page.evaluate(() => { + (window as any).onPayPalApprove?.({ orderID: "PAYPAL-ORDER-123" }); + }); + + await expect(page.getByText("Order confirmed")).toBeVisible(); +}); +``` + +### Payment Fixture + +```typescript +// fixtures/payment.fixture.ts +type PaymentFixtures = { + mockStripe: (options?: { failPayment?: boolean }) => Promise; +}; + +export const test = base.extend({ + mockStripe: async ({ page }, use) => { + await use(async (options = {}) => { + await page.addInitScript( + ([shouldFail]) => { + (window as any).Stripe = () => ({ + elements: () => ({ + create: () => ({ + mount: () => {}, + on: (event: string, handler: Function) => { + if (event === "ready") setTimeout(handler, 100); + }, + destroy: () => {}, + }), + }), + confirmCardPayment: async () => { + if (shouldFail) { + return { error: { message: "Card declined" } }; + } + return { paymentIntent: { status: "succeeded" } }; + }, + }); + }, + [options.failPayment], + ); + }); + }, +}); + +// Usage +test("handles declined card", async ({ page, mockStripe }) => { + await mockStripe({ failPayment: true }); + + await page.goto("/checkout"); + await page.getByRole("button", { name: "Pay" }).click(); + + await expect(page.getByText("Card declined")).toBeVisible(); +}); +``` + +## Email Verification + +### Mock Email API + +```typescript +test("email verification flow", async ({ page, request }) => { + let verificationToken: string; + + // Capture the verification email + await page.route("**/api/send-verification", async (route) => { + const body = route.request().postDataJSON(); + verificationToken = `mock-token-${Date.now()}`; + + // Don't actually send email, just store token + route.fulfill({ + json: { sent: true, messageId: "msg-123" }, + }); + }); + + // Mock token verification + await page.route("**/api/verify-email**", (route) => { + const url = new URL(route.request().url()); + const token = url.searchParams.get("token"); + + if (token === verificationToken) { + route.fulfill({ json: { verified: true } }); + } else { + route.fulfill({ status: 400, json: { error: "Invalid token" } }); + } + }); + + await page.goto("/signup"); + await page.getByLabel("Email").fill("test@example.com"); + await page.getByRole("button", { name: "Sign Up" }).click(); + + await expect(page.getByText("Check your email")).toBeVisible(); + + // Simulate clicking email link + await page.goto(`/verify?token=${verificationToken}`); + + await expect(page.getByText("Email verified")).toBeVisible(); +}); +``` + +### Use Mailinator/Temp Mail + +```typescript +// fixtures/email.fixture.ts +type EmailFixtures = { + getVerificationEmail: (inbox: string) => Promise<{ link: string }>; +}; + +export const test = base.extend({ + getVerificationEmail: async ({ request }, use) => { + await use(async (inbox) => { + // Poll Mailinator API for new email + const response = await request.get( + `https://api.mailinator.com/v2/domains/public/inboxes/${inbox}`, + { + headers: { + Authorization: `Bearer ${process.env.MAILINATOR_API_KEY}`, + }, + }, + ); + + const messages = await response.json(); + const latest = messages.msgs[0]; + + // Get full message + const msgResponse = await request.get( + `https://api.mailinator.com/v2/domains/public/inboxes/${inbox}/messages/${latest.id}`, + { + headers: { + Authorization: `Bearer ${process.env.MAILINATOR_API_KEY}`, + }, + }, + ); + + const message = await msgResponse.json(); + + // Extract verification link from HTML + const linkMatch = message.parts[0].body.match( + /href="([^"]*verify[^"]*)"/, + ); + return { link: linkMatch?.[1] || "" }; + }); + }, +}); +``` + +## SMS Verification + +### Mock SMS API + +```typescript +test("SMS verification", async ({ page }) => { + let smsCode: string; + + // Capture SMS send + await page.route("**/api/send-sms", (route) => { + smsCode = Math.random().toString().slice(2, 8); // 6-digit code + + route.fulfill({ + json: { sent: true, messageId: "sms-123" }, + }); + }); + + // Mock code verification + await page.route("**/api/verify-sms", (route) => { + const body = route.request().postDataJSON(); + + if (body.code === smsCode) { + route.fulfill({ json: { verified: true } }); + } else { + route.fulfill({ status: 400, json: { error: "Invalid code" } }); + } + }); + + await page.goto("/verify-phone"); + await page.getByLabel("Phone").fill("+1234567890"); + await page.getByRole("button", { name: "Send Code" }).click(); + + // Enter the code + await page.getByLabel("Verification Code").fill(smsCode); + await page.getByRole("button", { name: "Verify" }).click(); + + await expect(page.getByText("Phone verified")).toBeVisible(); +}); +``` + +## Analytics & Tracking + +### Block Analytics in Tests + +```typescript +test.beforeEach(async ({ page }) => { + // Block all analytics/tracking + await page.route( + /google-analytics|googletagmanager|facebook|hotjar|segment|mixpanel|amplitude/, + (route) => route.abort(), + ); +}); +``` + +### Mock Analytics for Verification + +```typescript +test("tracks purchase event", async ({ page }) => { + const analyticsEvents: any[] = []; + + // Capture analytics calls + await page.route("**/api/analytics/**", (route) => { + analyticsEvents.push(route.request().postDataJSON()); + route.fulfill({ status: 200 }); + }); + + // Mock analytics SDK + await page.addInitScript(() => { + (window as any).analytics = { + track: (event: string, props: any) => { + fetch("/api/analytics/track", { + method: "POST", + body: JSON.stringify({ event, props }), + }); + }, + }; + }); + + await page.goto("/checkout"); + await page.getByRole("button", { name: "Complete Purchase" }).click(); + + // Verify analytics event was sent + expect(analyticsEvents).toContainEqual( + expect.objectContaining({ + event: "Purchase Completed", + props: expect.objectContaining({ amount: expect.any(Number) }), + }), + ); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ------------------------- | ------------------------------ | ----------------------- | +| Using real OAuth in tests | Slow, needs credentials, flaky | Mock OAuth endpoints | +| Real payment processing | Charges real money, slow | Use test mode or mock | +| Waiting for real emails | Very slow, unreliable | Mock email API | +| Not mocking analytics | Pollutes analytics data | Block or mock analytics | + +## Related References + +- **Network Mocking**: See [network-advanced.md](network-advanced.md) for route patterns +- **Authentication**: See [fixtures-hooks.md](../core/fixtures-hooks.md) for auth patterns diff --git a/external/develop/devops/playwright-best-practices/architecture/pom-vs-fixtures.md b/external/develop/devops/playwright-best-practices/architecture/pom-vs-fixtures.md new file mode 100644 index 0000000..eafb06f --- /dev/null +++ b/external/develop/devops/playwright-best-practices/architecture/pom-vs-fixtures.md @@ -0,0 +1,363 @@ +# Organizing Reusable Test Code + +## Table of Contents + +1. [Pattern Comparison](#pattern-comparison) +2. [Selection Flowchart](#selection-flowchart) +3. [Page Objects](#page-objects) +4. [Custom Fixtures](#custom-fixtures) +5. [Helper Functions](#helper-functions) +6. [Combined Project Structure](#combined-project-structure) +7. [Anti-Patterns](#anti-patterns) + +Use all three patterns together. Most projects benefit from a hybrid approach: + +- **Page objects** for UI interaction (pages/components with 5+ interactions) +- **Custom fixtures** for test infrastructure (auth state, database, API clients, anything with lifecycle) +- **Helper functions** for stateless utilities (generate data, format values, simple waits) + +If only using one pattern, choose **custom fixtures** — they handle setup/teardown, compose well, and Playwright is built around them. + +## Pattern Comparison + +| Aspect | Page Objects | Custom Fixtures | Helper Functions | +|---|---|---|---| +| **Purpose** | Encapsulate UI interactions | Provide resources with setup/teardown | Stateless utilities | +| **Lifecycle** | Manual (constructor/methods) | Built-in (`use()` with automatic teardown) | None | +| **Composability** | Constructor injection or fixture wiring | Depend on other fixtures | Call other functions | +| **Best for** | Pages with many reused interactions | Resources needing setup AND teardown | Simple logic with no side effects | + +## Selection Flowchart + +```text +What kind of reusable code? +| ++-- Interacts with browser page/component? +| | +| +-- Has 5+ interactions (fill, click, navigate, assert)? +| | +-- YES: Used in 3+ test files? +| | | +-- YES --> PAGE OBJECT +| | | +-- NO --> Inline or small helper +| | +-- NO --> HELPER FUNCTION +| | +| +-- Needs setup before AND cleanup after test? +| +-- YES --> CUSTOM FIXTURE +| +-- NO --> PAGE OBJECT method or HELPER +| ++-- Manages resource with lifecycle (create/destroy)? +| +-- Examples: auth state, DB connection, API client, test user +| +-- YES --> CUSTOM FIXTURE (always) +| ++-- Stateless utility? (no browser, no side effects) +| +-- Examples: random email, format date, build URL, parse response +| +-- YES --> HELPER FUNCTION +| ++-- Not sure? + +-- Start with HELPER FUNCTION + +-- Promote to PAGE OBJECT when interactions grow + +-- Promote to FIXTURE when lifecycle needed +``` + +## Page Objects + +Best for pages/components with 5+ interactions appearing in 3+ test files. + +```typescript +// page-objects/booking.page.ts +import { type Page, type Locator, expect } from '@playwright/test'; + +export class BookingPage { + readonly page: Page; + readonly dateField: Locator; + readonly guestCount: Locator; + readonly roomType: Locator; + readonly reserveBtn: Locator; + readonly totalPrice: Locator; + + constructor(page: Page) { + this.page = page; + this.dateField = page.getByLabel('Check-in date'); + this.guestCount = page.getByLabel('Guests'); + this.roomType = page.getByLabel('Room type'); + this.reserveBtn = page.getByRole('button', { name: 'Reserve' }); + this.totalPrice = page.getByTestId('total-price'); + } + + async goto() { + await this.page.goto('/booking'); + } + + async fillDetails(opts: { date: string; guests: number; room: string }) { + await this.dateField.fill(opts.date); + await this.guestCount.fill(String(opts.guests)); + await this.roomType.selectOption(opts.room); + } + + async reserve() { + await this.reserveBtn.click(); + await this.page.waitForURL('**/confirmation'); + } + + async expectPrice(amount: string) { + await expect(this.totalPrice).toHaveText(amount); + } +} +``` + +```typescript +// tests/booking/reservation.spec.ts +import { test, expect } from '@playwright/test'; +import { BookingPage } from '../page-objects/booking.page'; + +test('complete reservation with standard room', async ({ page }) => { + const booking = new BookingPage(page); + await booking.goto(); + await booking.fillDetails({ date: '2026-03-15', guests: 2, room: 'standard' }); + await booking.reserve(); + await expect(page.getByText('Reservation confirmed')).toBeVisible(); +}); +``` + +**Page object principles:** +- One class per logical page/component, not per URL +- Constructor takes `Page` +- Locators as `readonly` properties in constructor +- Methods represent user intent (`reserve`, `fillDetails`), not low-level clicks +- Navigation methods (`goto`) belong on the page object + +## Custom Fixtures + +Best for resources needing setup before and teardown after tests — auth state, database connections, API clients, test users. + +```typescript +// fixtures/base.fixture.ts +import { test as base, expect } from '@playwright/test'; +import { BookingPage } from '../page-objects/booking.page'; +import { generateMember } from '../helpers/data'; + +type Fixtures = { + bookingPage: BookingPage; + member: { email: string; password: string; id: string }; + loggedInPage: import('@playwright/test').Page; +}; + +export const test = base.extend({ + bookingPage: async ({ page }, use) => { + await use(new BookingPage(page)); + }, + + member: async ({ request }, use) => { + const data = generateMember(); + const res = await request.post('/api/test/members', { data }); + const member = await res.json(); + await use(member); + await request.delete(`/api/test/members/${member.id}`); + }, + + loggedInPage: async ({ page, member }, use) => { + await page.goto('/login'); + await page.getByLabel('Email').fill(member.email); + await page.getByLabel('Password').fill(member.password); + await page.getByRole('button', { name: 'Sign in' }).click(); + await expect(page).toHaveURL('/dashboard'); + await use(page); + }, +}); + +export { expect } from '@playwright/test'; +``` + +```typescript +// tests/dashboard/overview.spec.ts +import { test, expect } from '../../fixtures/base.fixture'; + +test('member sees dashboard widgets', async ({ loggedInPage }) => { + await expect(loggedInPage.getByRole('heading', { name: 'Dashboard' })).toBeVisible(); + await expect(loggedInPage.getByTestId('stats-widget')).toBeVisible(); +}); + +test('new member sees welcome prompt', async ({ loggedInPage, member }) => { + await expect(loggedInPage.getByText(`Welcome, ${member.email}`)).toBeVisible(); +}); +``` + +**Fixture principles:** +- Use `test.extend()` — never module-level variables +- `use()` callback separates setup from teardown +- Teardown runs even if test fails +- Fixtures compose: one can depend on another +- Fixtures are lazy: created only when requested +- Wrap page objects in fixtures for lifecycle management + +## Helper Functions + +Best for stateless utilities — generating test data, formatting values, building URLs, parsing responses. + +```typescript +// helpers/data.ts +import { randomUUID } from 'node:crypto'; + +export function generateEmail(prefix = 'user'): string { + return `${prefix}-${Date.now()}-${randomUUID().slice(0, 8)}@test.local`; +} + +export function generateMember(overrides: Partial = {}): Member { + return { + email: generateEmail(), + password: 'SecurePass456!', + name: 'Test Member', + ...overrides, + }; +} + +interface Member { + email: string; + password: string; + name: string; +} + +export function formatPrice(cents: number): string { + return `$${(cents / 100).toFixed(2)}`; +} +``` + +```typescript +// helpers/assertions.ts +import { type Page, expect } from '@playwright/test'; + +export async function expectNotification(page: Page, message: string): Promise { + const notification = page.getByRole('alert').filter({ hasText: message }); + await expect(notification).toBeVisible(); + await expect(notification).toBeHidden({ timeout: 10000 }); +} +``` + +```typescript +// tests/settings/account.spec.ts +import { test, expect } from '@playwright/test'; +import { generateEmail } from '../../helpers/data'; +import { expectNotification } from '../../helpers/assertions'; + +test('update account email', async ({ page }) => { + const newEmail = generateEmail('updated'); + await page.goto('/settings/account'); + await page.getByLabel('Email').fill(newEmail); + await page.getByRole('button', { name: 'Save' }).click(); + await expectNotification(page, 'Account updated'); + await expect(page.getByLabel('Email')).toHaveValue(newEmail); +}); +``` + +**Helper principles:** +- Pure functions with no side effects +- No browser state — take `page` as parameter if needed +- Promote to fixture if setup/teardown needed +- Promote to page object if many page interactions grow +- Keep small and focused + +## Combined Project Structure + +```text +tests/ ++-- fixtures/ +| +-- auth.fixture.ts +| +-- db.fixture.ts +| +-- base.fixture.ts ++-- page-objects/ +| +-- login.page.ts +| +-- booking.page.ts +| +-- components/ +| +-- data-table.component.ts ++-- helpers/ +| +-- data.ts +| +-- assertions.ts ++-- e2e/ +| +-- auth/ +| | +-- login.spec.ts +| +-- booking/ +| +-- reservation.spec.ts +playwright.config.ts +``` + +**Layer responsibilities:** + +| Layer | Pattern | Responsibility | +|---|---|---| +| **Test file** | `test()` | Describes behavior, orchestrates layers | +| **Fixtures** | `test.extend()` | Resource lifecycle — setup, provide, teardown | +| **Page objects** | Classes | UI interaction — navigation, actions, locators | +| **Helpers** | Functions | Utilities — data generation, formatting, assertions | + +## Anti-Patterns + +### Page object managing resources + +```typescript +// BAD: page object handling API calls and database +class LoginPage { + async createUser() { /* API call */ } + async deleteUser() { /* API call */ } + async signIn(email: string, password: string) { /* UI */ } +} +``` + +Resource lifecycle belongs in fixtures where teardown is guaranteed. Keep only `signIn` in the page object. + +### Locator-only page objects + +```typescript +// BAD: no methods, just locators +class LoginPage { + emailInput = this.page.getByLabel('Email'); + passwordInput = this.page.getByLabel('Password'); + submitBtn = this.page.getByRole('button', { name: 'Sign in' }); + constructor(private page: Page) {} +} +``` + +Add intent-revealing methods or skip the page object entirely. + +### Monolithic fixtures + +```typescript +// BAD: one fixture doing everything +test.extend({ + everything: async ({ page, request }, use) => { + const user = await createUser(request); + const products = await seedProducts(request, 50); + await setupPayment(request, user.id); + await page.goto('/dashboard'); + await use({ user, products, page }); + // massive teardown... + }, +}); +``` + +Break into small, composable fixtures. Each fixture does one thing. + +### Helpers with side effects + +```typescript +// BAD: module-level state +let createdUserId: string; + +export async function createTestUser(request: APIRequestContext) { + const res = await request.post('/api/users', { data: { email: 'test@example.com' } }); + const user = await res.json(); + createdUserId = user.id; // shared across tests! + return user; +} +``` + +Module-level state leaks between parallel tests. If it has side effects and needs cleanup, make it a fixture. + +### Over-abstracting simple operations + +```typescript +// BAD: helper for one-liner +export async function clickButton(page: Page, name: string) { + await page.getByRole('button', { name }).click(); +} +``` + +Only abstract when there is real duplication (3+ usages) or complexity (5+ interactions). diff --git a/external/develop/devops/playwright-best-practices/architecture/test-architecture.md b/external/develop/devops/playwright-best-practices/architecture/test-architecture.md new file mode 100644 index 0000000..28b6f6c --- /dev/null +++ b/external/develop/devops/playwright-best-practices/architecture/test-architecture.md @@ -0,0 +1,369 @@ +# Choosing Test Types: E2E, Component, or API + +## Table of Contents + +1. [Decision Matrix](#decision-matrix) +2. [API Tests](#api-tests) +3. [Component Tests](#component-tests) +4. [E2E Tests](#e2e-tests) +5. [Layering Test Types](#layering-test-types) +6. [Common Mistakes](#common-mistakes) +7. [Related](#related) + +> **When to use**: Deciding which test type to write for a feature. Ask: "What's the cheapest test that gives confidence this works?" + +## Decision Matrix + +| Scenario | Recommended Type | Rationale | +| --------------------------- | ---------------- | --------------------------------------------- | +| Login / auth flow | E2E | Cross-page, cookies, redirects, session state | +| Form submission | Component | Isolated validation logic, error states | +| CRUD operations | API | Data integrity matters more than UI | +| Search with results UI | Component + API | API for query logic; component for rendering | +| Cross-page navigation | E2E | Routing, history, deep linking | +| API error handling | API | Status codes, error shapes, edge cases | +| UI error feedback | Component | Toast, banner, inline error rendering | +| Accessibility | Component | ARIA roles, keyboard nav per-component | +| Responsive layout | Component | Viewport-specific rendering without full app | +| API contract validation | API | Response shapes, headers, auth | +| WebSocket/real-time | E2E | Requires full browser environment | +| Payment / checkout | E2E | Multi-step, third-party iframes | +| Onboarding wizard | E2E | Multi-step, state persists across pages | +| Widget behavior | Component | Toggle, accordion, date picker, modal | +| Permissions / authorization | API | Role-based access is backend logic | + +## API Tests + +**Ideal for**: + +- CRUD operations (create, read, update, delete) +- Input validation and error responses (400, 422) +- Permission and authorization checks +- Data integrity and business rules +- API contract verification +- Edge cases expensive to reproduce through UI +- Test data setup/teardown for E2E tests + +**Avoid for**: + +- Testing how errors display to users +- Browser-specific behavior (cookies, redirects) +- Visual layout or responsive design +- Flows requiring JavaScript execution or DOM interaction +- Third-party iframe interactions + +```typescript +import { test, expect } from "@playwright/test"; + +test.describe("Products API", () => { + let token: string; + + test.beforeAll(async ({ request }) => { + const res = await request.post("/api/auth/token", { + data: { email: "manager@shop.io", password: "mgr-secret" }, + }); + token = (await res.json()).accessToken; + }); + + test("creates product with valid payload", async ({ request }) => { + const res = await request.post("/api/products", { + headers: { Authorization: `Bearer ${token}` }, + data: { name: "Widget Pro", sku: "WGT-100", price: 29.99 }, + }); + + expect(res.status()).toBe(201); + const product = await res.json(); + expect(product).toMatchObject({ name: "Widget Pro", sku: "WGT-100" }); + expect(product).toHaveProperty("id"); + }); + + test("rejects duplicate SKU with 409", async ({ request }) => { + const res = await request.post("/api/products", { + headers: { Authorization: `Bearer ${token}` }, + data: { name: "Duplicate", sku: "WGT-100", price: 19.99 }, + }); + + expect(res.status()).toBe(409); + expect((await res.json()).message).toContain("already exists"); + }); + + test("returns 422 for missing required fields", async ({ request }) => { + const res = await request.post("/api/products", { + headers: { Authorization: `Bearer ${token}` }, + data: { name: "Incomplete" }, + }); + + expect(res.status()).toBe(422); + const err = await res.json(); + expect(err.errors).toContainEqual( + expect.objectContaining({ field: "sku" }) + ); + }); + + test("staff role cannot delete products", async ({ request }) => { + const staffLogin = await request.post("/api/auth/token", { + data: { email: "staff@shop.io", password: "staff-pass" }, + }); + const staffToken = (await staffLogin.json()).accessToken; + + const res = await request.delete("/api/products/123", { + headers: { Authorization: `Bearer ${staffToken}` }, + }); + + expect(res.status()).toBe(403); + }); + + test("lists products with pagination", async ({ request }) => { + const res = await request.get("/api/products", { + headers: { Authorization: `Bearer ${token}` }, + params: { page: "1", limit: "20" }, + }); + + expect(res.status()).toBe(200); + const body = await res.json(); + expect(body.items).toBeInstanceOf(Array); + expect(body.items.length).toBeLessThanOrEqual(20); + expect(body).toHaveProperty("totalCount"); + }); +}); +``` + +## Component Tests + +**Ideal for**: + +- Form validation (required fields, format rules, error messages) +- Interactive widgets (modals, dropdowns, accordions, date pickers) +- Conditional rendering (show/hide, loading states, empty states) +- Accessibility per-component (ARIA attributes, keyboard navigation) +- Responsive layout at different viewports +- Visual states (hover, focus, disabled, selected) + +**Avoid for**: + +- Testing routing or navigation between pages +- Flows requiring real cookies, sessions, or server-side state +- Data persistence or API contract validation +- Third-party iframe interactions +- Anything requiring multiple pages or browser contexts + +```typescript +import { test, expect } from "@playwright/experimental-ct-react"; +import { ContactForm } from "../src/components/ContactForm"; + +test.describe("ContactForm component", () => { + test("displays validation errors on empty submit", async ({ mount }) => { + const component = await mount( {}} />); + + await component.getByRole("button", { name: "Send message" }).click(); + + await expect(component.getByText("Name is required")).toBeVisible(); + await expect(component.getByText("Email is required")).toBeVisible(); + }); + + test("rejects malformed email", async ({ mount }) => { + const component = await mount( {}} />); + + await component.getByLabel("Name").fill("Alex"); + await component.getByLabel("Email").fill("invalid-email"); + await component.getByLabel("Message").fill("Hello"); + await component.getByRole("button", { name: "Send message" }).click(); + + await expect(component.getByText("Enter a valid email")).toBeVisible(); + }); + + test("invokes onSubmit with form data", async ({ mount }) => { + const submissions: Array<{ name: string; email: string; message: string }> = + []; + const component = await mount( + submissions.push(data)} /> + ); + + await component.getByLabel("Name").fill("Alex"); + await component.getByLabel("Email").fill("alex@company.org"); + await component.getByLabel("Message").fill("Inquiry about pricing"); + await component.getByRole("button", { name: "Send message" }).click(); + + expect(submissions).toHaveLength(1); + expect(submissions[0]).toEqual({ + name: "Alex", + email: "alex@company.org", + message: "Inquiry about pricing", + }); + }); + + test("disables button during submission", async ({ mount }) => { + const component = await mount( + {}} submitting={true} /> + ); + + await expect( + component.getByRole("button", { name: "Sending..." }) + ).toBeDisabled(); + }); + + test("associates labels with inputs for accessibility", async ({ mount }) => { + const component = await mount( {}} />); + + await expect( + component.getByRole("textbox", { name: "Name" }) + ).toBeVisible(); + await expect( + component.getByRole("textbox", { name: "Email" }) + ).toBeVisible(); + }); +}); +``` + +## E2E Tests + +**Ideal for**: + +- Critical user flows that generate revenue (checkout, signup) +- Authentication flows (login, SSO, MFA, password reset) +- Multi-page workflows where state carries across navigation +- Flows involving third-party iframes (payment widgets) +- Smoke tests validating the entire stack +- Real-time collaboration requiring multiple browser contexts + +**Avoid for**: + +- Testing every form validation permutation +- CRUD operations where UI is a thin wrapper +- Verifying individual component states +- Testing API response shapes or error codes +- Responsive layout at every breakpoint +- Edge cases that only affect the backend + +```typescript +import { test, expect } from "@playwright/test"; + +test.describe("subscription flow", () => { + test.beforeEach(async ({ page }) => { + await page.request.post("/api/test/seed-account", { + data: { plan: "free", email: "subscriber@demo.io" }, + }); + await page.goto("/account/upgrade"); + }); + + test("upgrades to premium plan", async ({ page }) => { + await test.step("select plan", async () => { + await expect( + page.getByRole("heading", { name: "Choose Your Plan" }) + ).toBeVisible(); + await page.getByRole("button", { name: "Select Premium" }).click(); + }); + + await test.step("enter billing details", async () => { + await page.getByLabel("Cardholder name").fill("Sam Johnson"); + await page.getByLabel("Billing address").fill("456 Oak Ave"); + await page.getByLabel("City").fill("Seattle"); + await page.getByRole("combobox", { name: "State" }).selectOption("WA"); + await page.getByLabel("Postal code").fill("98101"); + await page.getByRole("button", { name: "Continue" }).click(); + }); + + await test.step("complete payment", async () => { + const paymentFrame = page.frameLocator('iframe[title="Secure Payment"]'); + await paymentFrame.getByLabel("Card number").fill("5555555555554444"); + await paymentFrame.getByLabel("Expiry").fill("09/29"); + await paymentFrame.getByLabel("CVV").fill("456"); + await page.getByRole("button", { name: "Subscribe now" }).click(); + }); + + await test.step("verify success", async () => { + await page.waitForURL("**/account/subscription/success**"); + await expect( + page.getByRole("heading", { name: "Welcome to Premium" }) + ).toBeVisible(); + await expect(page.getByText(/Subscription #\d+/)).toBeVisible(); + }); + }); +}); +``` + +## Layering Test Types + +Effective test suites combine all three types. Example for an "inventory management" feature: + +### API Layer (60% of tests) + +Cover every backend logic permutation. Cheap to run and maintain. + +``` +tests/api/inventory.spec.ts + - creates item with valid data (201) + - rejects duplicate SKU (409) + - rejects invalid quantity format (422) + - rejects missing required fields (422) + - warehouse-staff cannot delete items (403) + - unauthenticated request returns 401 + - lists items with pagination + - filters items by category + - updates item stock level + - archives an item + - prevents archiving items with pending orders +``` + +### Component Layer (30% of tests) + +Cover every visual state and interaction. + +``` +tests/components/InventoryForm.spec.tsx + - shows validation errors on empty submit + - shows inline error for invalid SKU format + - disables submit while saving + - calls onSubmit with form data + - resets form after successful save + +tests/components/InventoryTable.spec.tsx + - renders item rows from props + - shows empty state when no items + - handles archive confirmation modal + - sorts by column header click + - shows stock level badges with correct colors +``` + +### E2E Layer (10% of tests) + +Cover only critical paths proving full stack works. + +``` +tests/e2e/inventory.spec.ts + - manager creates item and sees it in list + - manager updates item stock level + - warehouse-staff cannot access admin settings +``` + +### Execution Profile + +For this feature: + +- **11 API tests** — ~2 seconds total, no browser +- **10 component tests** — ~5 seconds total, real browser but no server +- **3 E2E tests** — ~15 seconds total, full stack + +Total: 24 tests, ~22 seconds. API tests catch most regressions. Component tests catch UI bugs. E2E tests prove wiring works. If E2E fails but API and component pass, the problem is in integration (routing, state management, API client). + +## Common Mistakes + +| Anti-Pattern | Problem | Better Approach | +| ----------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------- | +| E2E for every validation rule | 30-second browser test for something API covers in 200ms | API test for validation, one component test for error display | +| No API tests, all E2E | Slow suite, flaky from UI timing, hard to diagnose | API tests for data/logic, E2E for critical paths only | +| Component tests mocking everything | Tests pass but app broken because mocks drift | Mock only external boundaries; API tests verify real contracts | +| Same assertion in API, component, AND E2E | Triple maintenance cost | Each layer tests what it uniquely verifies | +| E2E creating test data via UI | 2-minute test where 90 seconds is setup | Seed via API in `beforeEach`, test actual flow | +| Testing third-party behavior | Testing that Stripe validates cards (Stripe's job) | Mock Stripe; trust their contract | +| Skipping API layer | Can't tell if bug is frontend or backend | API tests isolate backend; component tests isolate frontend | +| One giant E2E for entire feature | 5-minute test failing somewhere with no clear cause | Focused E2E per critical path; use `test.step()` | + +## Related + +- [test-suite-structure.md](../core/test-suite-structure.md) — file structure and naming +- [api-testing.md](../testing-patterns/api-testing.md) — Playwright's `request` API for HTTP testing +- [component-testing.md](../testing-patterns/component-testing.md) — setting up component tests +- [authentication.md](../advanced/authentication.md) — auth flow patterns with `storageState` +- [when-to-mock.md](when-to-mock.md) — when to mock vs hit real services +- [pom-vs-fixtures.md](pom-vs-fixtures.md) — organizing shared test logic diff --git a/external/develop/devops/playwright-best-practices/architecture/when-to-mock.md b/external/develop/devops/playwright-best-practices/architecture/when-to-mock.md new file mode 100644 index 0000000..d5d5705 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/architecture/when-to-mock.md @@ -0,0 +1,383 @@ +# Mocking Strategy: Real vs Mock Services + +## Table of Contents + +1. [Core Principle](#core-principle) +2. [Decision Matrix](#decision-matrix) +3. [Decision Flowchart](#decision-flowchart) +4. [Mocking Techniques](#mocking-techniques) +5. [Real Service Strategies](#real-service-strategies) +6. [Hybrid Approach: Fixture-Based Mock Control](#hybrid-approach-fixture-based-mock-control) +7. [Validating Mock Accuracy](#validating-mock-accuracy) +8. [Anti-Patterns](#anti-patterns) + +> **When to use**: Deciding whether to mock API calls, intercept network requests, or hit real services in Playwright tests. + +## Core Principle + +**Mock at the boundary, test your stack end-to-end.** Mock third-party services you don't own (payment gateways, email providers, OAuth). Never mock your own frontend-to-backend communication. Tests should prove YOUR code works, not that third-party APIs are available. + +## Decision Matrix + +| Scenario | Mock? | Strategy | +| --- | --- | --- | +| Your own REST/GraphQL API | Never | Hit real API against staging or local dev | +| Your database (through your API) | Never | Seed via API or fixtures | +| Authentication (your auth system) | Mostly no | Use `storageState` to skip login in most tests | +| Stripe / payment gateway | Always | `route.fulfill()` with expected responses | +| SendGrid / email service | Always | Mock the API call, verify request payload | +| OAuth providers (Google, GitHub) | Always | Mock token exchange, test your callback handler | +| Analytics (Segment, Mixpanel) | Always | `route.abort()` or `route.fulfill()` | +| Maps / geocoding APIs | Always | Mock with static responses | +| Feature flags (LaunchDarkly) | Usually | Mock to force specific flag states | +| CDN / static assets | Never | Let them load normally | +| Flaky external dependency | CI: mock, local: real | Conditional mocking based on environment | +| Slow external dependency | Dev: mock, nightly: real | Separate test projects in config | + +## Decision Flowchart + +```text +Is this service part of YOUR codebase? +├── YES → Do NOT mock. Test the real integration. +│ ├── Is it slow? → Optimize the service, not the test. +│ └── Is it flaky? → Fix the service. Flaky infra is a bug. +└── NO → It's a third-party service. + ├── Is it paid per call? → ALWAYS mock. + ├── Is it rate-limited? → ALWAYS mock. + ├── Is it slow or unreliable? → ALWAYS mock. + └── Is it a complex multi-step flow? → Mock with HAR recording. +``` + +## Mocking Techniques + +### Blocking Unwanted Requests + +Block third-party scripts that slow tests and add no coverage: + +```typescript +test.beforeEach(async ({ page }) => { + await page.route('**/{analytics,tracking,segment,hotjar}.{com,io}/**', (route) => { + route.abort(); + }); +}); + +test('dashboard renders without tracking scripts', async ({ page }) => { + await page.goto('/dashboard'); + await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible(); +}); +``` + +### Full Mock (route.fulfill) + +Completely replace a third-party API response: + +```typescript +test('order flow with mocked payment service', async ({ page }) => { + await page.route('**/api/charge', (route) => { + route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + transactionId: 'txn_mock_abc', + status: 'completed', + }), + }); + }); + + await page.goto('/order/confirm'); + await page.getByRole('button', { name: 'Complete Purchase' }).click(); + await expect(page.getByText('Order confirmed')).toBeVisible(); +}); + +test('display error on payment decline', async ({ page }) => { + await page.route('**/api/charge', (route) => { + route.fulfill({ + status: 402, + contentType: 'application/json', + body: JSON.stringify({ + error: { code: 'insufficient_funds', message: 'Card declined.' }, + }), + }); + }); + + await page.goto('/order/confirm'); + await page.getByRole('button', { name: 'Complete Purchase' }).click(); + await expect(page.getByRole('alert')).toContainText('Card declined'); +}); +``` + +### Partial Mock (Modify Responses) + +Let the real API call happen but tweak the response: + +```typescript +test('display low inventory warning', async ({ page }) => { + await page.route('**/api/inventory/*', async (route) => { + const response = await route.fetch(); + const data = await response.json(); + + data.quantity = 1; + data.lowStock = true; + + await route.fulfill({ + response, + body: JSON.stringify(data), + }); + }); + + await page.goto('/products/widget-pro'); + await expect(page.getByText('Only 1 remaining')).toBeVisible(); +}); + +test('inject test notification into real response', async ({ page }) => { + await page.route('**/api/alerts', async (route) => { + const response = await route.fetch(); + const data = await response.json(); + + data.items.push({ + id: 'test-alert', + text: 'Report generated', + category: 'info', + }); + + await route.fulfill({ + response, + body: JSON.stringify(data), + }); + }); + + await page.goto('/home'); + await expect(page.getByText('Report generated')).toBeVisible(); +}); +``` + +### Record and Replay (HAR Files) + +For complex API sequences (OAuth flows, multi-step wizards): + +**Recording:** + +```typescript +test('capture API traffic for admin panel', async ({ page }) => { + await page.routeFromHAR('tests/fixtures/admin-panel.har', { + url: '**/api/**', + update: true, + }); + + await page.goto('/admin'); + await page.getByRole('tab', { name: 'Reports' }).click(); + await page.getByRole('tab', { name: 'Settings' }).click(); +}); +``` + +**Replaying:** + +```typescript +test('admin panel loads with recorded data', async ({ page }) => { + await page.routeFromHAR('tests/fixtures/admin-panel.har', { + url: '**/api/**', + update: false, + }); + + await page.goto('/admin'); + await expect(page.getByRole('heading', { name: 'Reports' })).toBeVisible(); +}); +``` + +**HAR maintenance:** + +- Record against a known-good staging environment +- Commit `.har` files to version control +- Re-record when APIs change +- Scope HAR to specific URL patterns + +## Real Service Strategies + +### Local Dev Server + +```typescript +// playwright.config.ts +export default defineConfig({ + webServer: { + command: 'npm run dev', + url: 'http://localhost:3000', + reuseExistingServer: !process.env.CI, + timeout: 30_000, + }, + use: { + baseURL: 'http://localhost:3000', + }, +}); +``` + +### Staging Environment + +```typescript +// playwright.config.ts +export default defineConfig({ + use: { + baseURL: process.env.CI + ? 'https://staging.example.com' + : 'http://localhost:3000', + }, +}); +``` + +### Test Containers + +```typescript +// playwright.config.ts +export default defineConfig({ + webServer: { + command: 'docker compose -f docker-compose.test.yml up --wait', + url: 'http://localhost:3000/health', + reuseExistingServer: !process.env.CI, + timeout: 120_000, + }, + globalTeardown: './tests/global-teardown.ts', +}); +``` + +```typescript +// tests/global-teardown.ts +import { execSync } from 'child_process'; + +export default function globalTeardown() { + if (process.env.CI) { + execSync('docker compose -f docker-compose.test.yml down -v'); + } +} +``` + +## Hybrid Approach: Fixture-Based Mock Control + +Create fixtures that let individual tests opt into mocking specific services: + +```typescript +// tests/fixtures/service-mocks.ts +import { test as base } from '@playwright/test'; + +type MockConfig = { + mockPayments: boolean; + mockNotifications: boolean; + mockAnalytics: boolean; +}; + +export const test = base.extend({ + mockPayments: [true, { option: true }], + mockNotifications: [true, { option: true }], + mockAnalytics: [true, { option: true }], + + page: async ({ page, mockPayments, mockNotifications, mockAnalytics }, use) => { + if (mockPayments) { + await page.route('**/api/billing/**', (route) => { + route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ status: 'paid', id: 'inv_mock_789' }), + }); + }); + } + + if (mockNotifications) { + await page.route('**/api/notify', (route) => { + route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ delivered: true }), + }); + }); + } + + if (mockAnalytics) { + await page.route('**/{segment,mixpanel,amplitude}.**/**', (route) => { + route.abort(); + }); + } + + await use(page); + }, +}); + +export { expect } from '@playwright/test'; +``` + +```typescript +// tests/billing.spec.ts +import { test, expect } from './fixtures/service-mocks'; + +test('subscription renewal sends notification', async ({ page }) => { + await page.goto('/account/billing'); + await page.getByRole('button', { name: 'Renew Now' }).click(); + await expect(page.getByText('Subscription renewed')).toBeVisible(); +}); + +test.describe('integration suite', () => { + test.use({ mockPayments: false }); + + test('real billing flow against test gateway', async ({ page }) => { + await page.goto('/account/billing'); + await page.getByRole('button', { name: 'Renew Now' }).click(); + await expect(page.getByText('Subscription renewed')).toBeVisible(); + }); +}); +``` + +### Environment-Based Test Projects + +```typescript +// playwright.config.ts +export default defineConfig({ + projects: [ + { + name: 'ci-fast', + testMatch: '**/*.spec.ts', + use: { baseURL: 'http://localhost:3000' }, + }, + { + name: 'nightly-full', + testMatch: '**/*.integration.spec.ts', + use: { baseURL: 'https://staging.example.com' }, + timeout: 120_000, + }, + ], +}); +``` + +## Validating Mock Accuracy + +Guard against mock drift from real APIs: + +```typescript +test.describe('contract validation', () => { + test('billing mock matches real API shape', async ({ request }) => { + const realResponse = await request.post('/api/billing/charge', { + data: { amount: 5000, currency: 'usd' }, + }); + const realBody = await realResponse.json(); + + const mockBody = { + status: 'paid', + id: 'inv_mock_789', + }; + + expect(Object.keys(mockBody).sort()).toEqual(Object.keys(realBody).sort()); + + for (const key of Object.keys(mockBody)) { + expect(typeof mockBody[key]).toBe(typeof realBody[key]); + } + }); +}); +``` + +## Anti-Patterns + +| Don't Do This | Problem | Do This Instead | +| --- | --- | --- | +| Mock your own API | Tests pass, app breaks. Zero integration coverage. | Hit your real API. Mock only third-party services. | +| Mock everything for speed | You test a fiction. Frontend and backend may be incompatible. | Mock only external boundaries. | +| Never mock anything | Tests are slow, flaky, fail when third parties have outages. | Mock third-party services. | +| Use outdated mocks | Mock returns different shape than real API. | Run contract validation tests. Re-record HAR files regularly. | +| Mock with `page.evaluate()` to stub fetch | Fragile, doesn't survive navigation. | Use `page.route()` which intercepts at network layer. | +| Copy-paste mocks across files | One API change requires updating many files. | Centralize mocks in fixtures. | +| Block all network and whitelist | Extremely brittle. Every new endpoint requires update. | Allow all by default. Selectively mock third-party services. | diff --git a/external/develop/devops/playwright-best-practices/browser-apis/browser-apis.md b/external/develop/devops/playwright-best-practices/browser-apis/browser-apis.md new file mode 100644 index 0000000..cc4c269 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/browser-apis/browser-apis.md @@ -0,0 +1,391 @@ +# Browser APIs: Geolocation, Permissions & More + +## Table of Contents + +1. [Geolocation](#geolocation) +2. [Permissions](#permissions) +3. [Clipboard](#clipboard) +4. [Notifications](#notifications) +5. [Camera & Microphone](#camera--microphone) + +## Geolocation + +### Mock Location + +```typescript +test("shows nearby stores", async ({ context }) => { + // Grant permission and set location + await context.grantPermissions(["geolocation"]); + await context.setGeolocation({ latitude: 37.7749, longitude: -122.4194 }); // San Francisco + + const page = await context.newPage(); + await page.goto("/store-finder"); + await page.getByRole("button", { name: "Find Nearby" }).click(); + + await expect(page.getByText("San Francisco")).toBeVisible(); +}); +``` + +### Geolocation Fixture + +```typescript +// fixtures/geolocation.fixture.ts +import { test as base } from "@playwright/test"; + +type Coordinates = { latitude: number; longitude: number; accuracy?: number }; + +type GeoFixtures = { + setLocation: (coords: Coordinates) => Promise; +}; + +export const test = base.extend({ + setLocation: async ({ context }, use) => { + await context.grantPermissions(["geolocation"]); + + await use(async (coords) => { + await context.setGeolocation({ + latitude: coords.latitude, + longitude: coords.longitude, + accuracy: coords.accuracy ?? 100, + }); + }); + }, +}); + +// Usage +test("delivery zone check", async ({ page, setLocation }) => { + await setLocation({ latitude: 40.7128, longitude: -74.006 }); // NYC + + await page.goto("/delivery"); + + await expect(page.getByText("Delivery available")).toBeVisible(); +}); +``` + +### Test Location Changes + +```typescript +test("tracks location updates", async ({ context }) => { + await context.grantPermissions(["geolocation"]); + + const page = await context.newPage(); + await page.goto("/tracking"); + + // Initial location + await context.setGeolocation({ latitude: 37.7749, longitude: -122.4194 }); + await page.getByRole("button", { name: "Start Tracking" }).click(); + + await expect(page.getByTestId("location")).toContainText("37.7749"); + + // Move to new location + await context.setGeolocation({ latitude: 37.8044, longitude: -122.2712 }); + + // Trigger location update + await page.evaluate(() => { + navigator.geolocation.getCurrentPosition(() => {}); + }); + + await expect(page.getByTestId("location")).toContainText("37.8044"); +}); +``` + +### Test Geolocation Denial + +```typescript +test("handles location denied", async ({ browser }) => { + // Create context without geolocation permission + const context = await browser.newContext({ + permissions: [], // No permissions + }); + + const page = await context.newPage(); + await page.goto("/store-finder"); + await page.getByRole("button", { name: "Find Nearby" }).click(); + + await expect(page.getByText("Location access denied")).toBeVisible(); + await expect(page.getByLabel("Enter ZIP code")).toBeVisible(); + + await context.close(); +}); +``` + +## Permissions + +### Grant Permissions + +```typescript +test("notifications with permission", async ({ context }) => { + await context.grantPermissions(["notifications"]); + + const page = await context.newPage(); + await page.goto("/alerts"); + + // Notification API should work + const permission = await page.evaluate(() => Notification.permission); + expect(permission).toBe("granted"); +}); +``` + +### Test Permission Denied + +```typescript +test("handles notification permission denied", async ({ browser }) => { + const context = await browser.newContext({ + permissions: [], // Deny all + }); + + const page = await context.newPage(); + await page.goto("/notifications"); + + await page.getByRole("button", { name: "Enable Notifications" }).click(); + + await expect(page.getByText("Please enable notifications")).toBeVisible(); + + await context.close(); +}); +``` + +### Multiple Permissions + +```typescript +test("video call with permissions", async ({ context }) => { + await context.grantPermissions(["camera", "microphone", "notifications"]); + + const page = await context.newPage(); + await page.goto("/video-call"); + + // All permissions should be granted + const permissions = await page.evaluate(async () => ({ + camera: await navigator.permissions.query({ + name: "camera" as PermissionName, + }), + microphone: await navigator.permissions.query({ + name: "microphone" as PermissionName, + }), + })); + + expect(permissions.camera.state).toBe("granted"); + expect(permissions.microphone.state).toBe("granted"); +}); +``` + +## Clipboard + +### Test Copy to Clipboard + +```typescript +test("copy button works", async ({ page, context }) => { + // Grant clipboard permissions + await context.grantPermissions(["clipboard-read", "clipboard-write"]); + + await page.goto("/share"); + + await page.getByRole("button", { name: "Copy Link" }).click(); + + // Read clipboard content + const clipboardContent = await page.evaluate(() => + navigator.clipboard.readText(), + ); + + expect(clipboardContent).toContain("https://example.com/share/"); +}); +``` + +### Test Paste from Clipboard + +```typescript +test("paste from clipboard", async ({ page, context }) => { + await context.grantPermissions(["clipboard-read", "clipboard-write"]); + + await page.goto("/editor"); + + // Write to clipboard + await page.evaluate(() => navigator.clipboard.writeText("Pasted content")); + + // Trigger paste + await page.getByLabel("Content").focus(); + await page.keyboard.press("Control+V"); + + await expect(page.getByLabel("Content")).toHaveValue("Pasted content"); +}); +``` + +### Clipboard Fixture + +```typescript +// fixtures/clipboard.fixture.ts +import { test as base } from "@playwright/test"; + +type ClipboardFixtures = { + clipboard: { + write: (text: string) => Promise; + read: () => Promise; + }; +}; + +export const test = base.extend({ + clipboard: async ({ page, context }, use) => { + await context.grantPermissions(["clipboard-read", "clipboard-write"]); + + await use({ + write: async (text) => { + await page.evaluate((t) => navigator.clipboard.writeText(t), text); + }, + read: async () => { + return page.evaluate(() => navigator.clipboard.readText()); + }, + }); + }, +}); +``` + +## Notifications + +### Mock Notification API + +```typescript +test("shows browser notification", async ({ page }) => { + const notifications: any[] = []; + + // Mock Notification constructor + await page.addInitScript(() => { + (window as any).__notifications = []; + (window as any).Notification = class { + constructor(title: string, options?: NotificationOptions) { + (window as any).__notifications.push({ title, ...options }); + } + static permission = "granted"; + static requestPermission = async () => "granted"; + }; + }); + + await page.goto("/alerts"); + await page.getByRole("button", { name: "Notify Me" }).click(); + + // Check notification was created + const created = await page.evaluate(() => (window as any).__notifications); + expect(created).toHaveLength(1); + expect(created[0].title).toBe("New Alert"); +}); +``` + +### Test Notification Click + +```typescript +test("notification click handler", async ({ page }) => { + await page.addInitScript(() => { + (window as any).Notification = class { + onclick: (() => void) | null = null; + constructor(title: string) { + // Simulate click after creation + setTimeout(() => this.onclick?.(), 100); + } + static permission = "granted"; + static requestPermission = async () => "granted"; + }; + }); + + await page.goto("/messages"); + await page.evaluate(() => { + new Notification("New Message"); + }); + + // Should navigate to messages when notification clicked + await expect(page).toHaveURL(/\/messages/); +}); +``` + +## Camera & Microphone + +### Mock Media Devices + +```typescript +test("video preview works", async ({ page, context }) => { + await context.grantPermissions(["camera"]); + + // Mock getUserMedia + await page.addInitScript(() => { + navigator.mediaDevices.getUserMedia = async () => { + const canvas = document.createElement("canvas"); + canvas.width = 640; + canvas.height = 480; + return canvas.captureStream(); + }; + }); + + await page.goto("/video-settings"); + await page.getByRole("button", { name: "Start Camera" }).click(); + + await expect(page.getByTestId("video-preview")).toBeVisible(); +}); +``` + +### Test Media Device Selection + +```typescript +test("switch camera", async ({ page }) => { + await page.addInitScript(() => { + navigator.mediaDevices.enumerateDevices = async () => + [ + { + deviceId: "cam1", + kind: "videoinput", + label: "Front Camera", + groupId: "1", + }, + { + deviceId: "cam2", + kind: "videoinput", + label: "Back Camera", + groupId: "2", + }, + ] as MediaDeviceInfo[]; + + navigator.mediaDevices.getUserMedia = async () => { + const canvas = document.createElement("canvas"); + return canvas.captureStream(); + }; + }); + + await page.goto("/camera"); + + // Should show camera options + await expect(page.getByRole("combobox", { name: "Camera" })).toBeVisible(); + await expect(page.getByText("Front Camera")).toBeVisible(); + await expect(page.getByText("Back Camera")).toBeVisible(); +}); +``` + +### Test Media Errors + +```typescript +test("handles camera access error", async ({ page }) => { + await page.addInitScript(() => { + navigator.mediaDevices.getUserMedia = async () => { + throw new DOMException("Permission denied", "NotAllowedError"); + }; + }); + + await page.goto("/video-call"); + await page.getByRole("button", { name: "Join Call" }).click(); + + await expect(page.getByText("Camera access denied")).toBeVisible(); + await expect( + page.getByRole("button", { name: "Join Audio Only" }), + ).toBeVisible(); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ----------------------------- | --------------------------------- | ----------------------------------- | +| Not granting permissions | Tests fail with permission errors | Use `context.grantPermissions()` | +| Testing real geolocation | Flaky, environment-dependent | Mock with `setGeolocation()` | +| Not testing permission denial | Misses error handling | Test both granted and denied states | +| Using real camera/mic | CI has no devices | Mock `getUserMedia` | + +## Related References + +- **Fixtures**: See [fixtures-hooks.md](../core/fixtures-hooks.md) for context fixtures +- **Mobile**: See [mobile-testing.md](../advanced/mobile-testing.md) for device emulation diff --git a/external/develop/devops/playwright-best-practices/browser-apis/iframes.md b/external/develop/devops/playwright-best-practices/browser-apis/iframes.md new file mode 100644 index 0000000..145e050 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/browser-apis/iframes.md @@ -0,0 +1,403 @@ +# iFrame Testing + +## Table of Contents + +1. [Basic iFrame Access](#basic-iframe-access) +2. [Cross-Origin iFrames](#cross-origin-iframes) +3. [Nested iFrames](#nested-iframes) +4. [Dynamic iFrames](#dynamic-iframes) +5. [iFrame Navigation](#iframe-navigation) +6. [Common Patterns](#common-patterns) + +## Basic iFrame Access + +### Using frameLocator + +```typescript +// Access iframe by selector +const frame = page.frameLocator("iframe#payment"); +await frame.getByRole("button", { name: "Pay" }).click(); + +// Access by name attribute +const namedFrame = page.frameLocator('iframe[name="checkout"]'); +await namedFrame.getByLabel("Card number").fill("4242424242424242"); + +// Access by title +const titledFrame = page.frameLocator('iframe[title="Payment Form"]'); + +// Access by src (partial match) +const srcFrame = page.frameLocator('iframe[src*="stripe.com"]'); +``` + +### Frame vs FrameLocator + +```typescript +// frameLocator - for locator-based operations (recommended) +const frameLocator = page.frameLocator("#my-iframe"); +await frameLocator.getByRole("button").click(); + +// frame() - for Frame object operations (navigation, evaluation) +const frame = page.frame({ name: "my-frame" }); +if (frame) { + await frame.goto("https://example.com"); + const title = await frame.title(); +} + +// Get all frames +const frames = page.frames(); +for (const f of frames) { + console.log("Frame URL:", f.url()); +} +``` + +### Waiting for iFrame Content + +```typescript +// Wait for iframe to load +const frame = page.frameLocator("#dynamic-iframe"); + +// Wait for element inside iframe +await expect(frame.getByRole("heading")).toBeVisible({ timeout: 10000 }); + +// Wait for iframe src to change +await page.waitForFunction(() => { + const iframe = document.querySelector("iframe#my-frame") as HTMLIFrameElement; + return iframe?.src.includes("loaded"); +}); +``` + +## Cross-Origin iFrames + +### Accessing Cross-Origin Content + +```typescript +// Cross-origin iframes work seamlessly with frameLocator +const thirdPartyFrame = page.frameLocator('iframe[src*="third-party.com"]'); + +// Interact with elements inside cross-origin iframe +await thirdPartyFrame.getByRole("textbox").fill("test@example.com"); +await thirdPartyFrame.getByRole("button", { name: "Submit" }).click(); + +// Wait for cross-origin iframe to be ready +await expect(thirdPartyFrame.locator("body")).toBeVisible(); +``` + +### Payment Provider iFrames (Stripe, PayPal) + +```typescript +test("Stripe payment iframe", async ({ page }) => { + await page.goto("/checkout"); + + // Stripe uses multiple iframes for each field + const cardFrame = page + .frameLocator('iframe[name*="__privateStripeFrame"]') + .first(); + + // Wait for Stripe to initialize + await expect(cardFrame.locator('[placeholder="Card number"]')).toBeVisible({ + timeout: 15000, + }); + + // Fill card details + await cardFrame + .locator('[placeholder="Card number"]') + .fill("4242424242424242"); + await cardFrame.locator('[placeholder="MM / YY"]').fill("12/30"); + await cardFrame.locator('[placeholder="CVC"]').fill("123"); +}); +``` + +### Handling OAuth in iFrames + +```typescript +test("OAuth iframe flow", async ({ page }) => { + await page.goto("/login"); + await page.getByRole("button", { name: "Sign in with Google" }).click(); + + // If OAuth opens in iframe instead of popup + const oauthFrame = page.frameLocator('iframe[src*="accounts.google.com"]'); + + // Wait for OAuth form + await expect(oauthFrame.getByLabel("Email")).toBeVisible({ timeout: 10000 }); + await oauthFrame.getByLabel("Email").fill("test@gmail.com"); +}); +``` + +## Nested iFrames + +### Accessing Nested Frames + +```typescript +// Parent iframe contains child iframe +const parentFrame = page.frameLocator("#outer-frame"); +const childFrame = parentFrame.frameLocator("#inner-frame"); + +// Interact with deeply nested content +await childFrame.getByRole("button", { name: "Submit" }).click(); + +// Multiple levels of nesting +const level1 = page.frameLocator("#level1"); +const level2 = level1.frameLocator("#level2"); +const level3 = level2.frameLocator("#level3"); +await level3.getByText("Deep content").click(); +``` + +### Finding Elements Across Frame Hierarchy + +```typescript +// Helper to search all frames for an element +async function findInAnyFrame( + page: Page, + selector: string, +): Promise { + // Check main page first + const mainCount = await page.locator(selector).count(); + if (mainCount > 0) return page.locator(selector); + + // Check all frames + for (const frame of page.frames()) { + const count = await frame.locator(selector).count(); + if (count > 0) { + return frame.locator(selector); + } + } + return null; +} + +test("find element in any frame", async ({ page }) => { + await page.goto("/complex-page"); + const element = await findInAnyFrame(page, '[data-testid="submit-btn"]'); + if (element) await element.click(); +}); +``` + +## Dynamic iFrames + +### iFrames Created at Runtime + +```typescript +test("handle dynamically created iframe", async ({ page }) => { + await page.goto("/dashboard"); + + // Click button that creates iframe + await page.getByRole("button", { name: "Open Widget" }).click(); + + // Wait for iframe to appear in DOM + await page.waitForSelector("iframe#widget-frame"); + + // Now access the frame + const widgetFrame = page.frameLocator("#widget-frame"); + await expect(widgetFrame.getByText("Widget Loaded")).toBeVisible(); +}); +``` + +### iFrames with Changing src + +```typescript +test("iframe src changes", async ({ page }) => { + await page.goto("/multi-step"); + + const frame = page.frameLocator("#step-frame"); + + // Step 1 + await expect(frame.getByText("Step 1")).toBeVisible(); + await frame.getByRole("button", { name: "Next" }).click(); + + // Wait for iframe to reload with new content + await expect(frame.getByText("Step 2")).toBeVisible({ timeout: 10000 }); + await frame.getByRole("button", { name: "Next" }).click(); + + // Step 3 + await expect(frame.getByText("Step 3")).toBeVisible({ timeout: 10000 }); +}); +``` + +### Lazy-Loaded iFrames + +```typescript +test("lazy loaded iframe", async ({ page }) => { + await page.goto("/page-with-lazy-iframe"); + + // Scroll to trigger lazy load + await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight)); + + // Wait for iframe to load + const lazyFrame = page.frameLocator("#lazy-iframe"); + await expect(lazyFrame.locator("body")).not.toBeEmpty({ timeout: 15000 }); + + // Interact with content + await lazyFrame.getByRole("button").click(); +}); +``` + +## iFrame Navigation + +### Navigating Within iFrame + +```typescript +test("iframe internal navigation", async ({ page }) => { + await page.goto("/app"); + + // Get frame object for navigation control + const frame = page.frame({ name: "content-frame" }); + if (!frame) throw new Error("Frame not found"); + + // Navigate within iframe + await frame.goto("https://embedded-app.com/page2"); + + // Wait for navigation + await frame.waitForURL("**/page2"); + + // Verify content + await expect(frame.getByRole("heading")).toHaveText("Page 2"); +}); +``` + +### Handling Frame Navigation Events + +```typescript +test("track iframe navigation", async ({ page }) => { + const navigations: string[] = []; + + // Listen to frame navigation + page.on("framenavigated", (frame) => { + if (frame.parentFrame()) { + // This is an iframe navigation + navigations.push(frame.url()); + } + }); + + await page.goto("/with-iframe"); + await page + .frameLocator("#nav-frame") + .getByRole("link", { name: "Page 2" }) + .click(); + + // Verify navigation occurred + expect(navigations.some((url) => url.includes("page2"))).toBe(true); +}); +``` + +## Common Patterns + +### iFrame Fixture + +```typescript +// fixtures.ts +import { test as base, FrameLocator } from "@playwright/test"; + +export const test = base.extend<{ paymentFrame: FrameLocator }>({ + paymentFrame: async ({ page }, use) => { + await page.goto("/checkout"); + + // Wait for payment iframe to be ready + const frame = page.frameLocator('iframe[src*="payment"]'); + await expect(frame.locator("body")).toBeVisible({ timeout: 15000 }); + + await use(frame); + }, +}); + +// test file +test("complete payment", async ({ paymentFrame }) => { + await paymentFrame.getByLabel("Card").fill("4242424242424242"); + await paymentFrame.getByRole("button", { name: "Pay" }).click(); +}); +``` + +### Debugging iFrame Issues + +```typescript +test("debug iframe content", async ({ page }) => { + await page.goto("/page-with-iframes"); + + // List all frames + console.log("All frames:"); + for (const frame of page.frames()) { + console.log(` - ${frame.name() || "(unnamed)"}: ${frame.url()}`); + } + + // Screenshot specific iframe content + const frame = page.frame({ name: "target-frame" }); + if (frame) { + const body = frame.locator("body"); + await body.screenshot({ path: "iframe-content.png" }); + } + + // Get iframe HTML for debugging + const frameContent = page.frameLocator("#my-frame"); + const html = await frameContent.locator("body").innerHTML(); + console.log("iFrame HTML:", html.substring(0, 500)); +}); +``` + +### Handling iFrame Load Failures + +```typescript +test("handle iframe load failure", async ({ page }) => { + await page.goto("/page-with-unreliable-iframe"); + + const frame = page.frameLocator("#unreliable-frame"); + + try { + // Try to interact with iframe content + await expect(frame.getByRole("button")).toBeVisible({ timeout: 5000 }); + await frame.getByRole("button").click(); + } catch (error) { + // Fallback: refresh iframe + await page.evaluate(() => { + const iframe = document.querySelector( + "#unreliable-frame", + ) as HTMLIFrameElement; + if (iframe) iframe.src = iframe.src; + }); + + // Retry + await expect(frame.getByRole("button")).toBeVisible({ timeout: 10000 }); + await frame.getByRole("button").click(); + } +}); +``` + +### Mocking iFrame Content + +```typescript +test("mock iframe response", async ({ page }) => { + // Intercept iframe src request + await page.route("**/embedded-widget**", (route) => { + route.fulfill({ + contentType: "text/html", + body: ` + + + +

Mocked Widget

+ + + + `, + }); + }); + + await page.goto("/page-with-widget"); + + const frame = page.frameLocator("#widget-frame"); + await expect(frame.getByRole("heading")).toHaveText("Mocked Widget"); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ------------------------------------- | --------------------------------- | -------------------------------------------------- | +| Using `page.frame()` for interactions | Less reliable than frameLocator | Use `page.frameLocator()` for element interactions | +| Hardcoding iframe index | Fragile if DOM order changes | Use name, id, or src attribute selectors | +| Not waiting for iframe load | Race conditions | Wait for element inside iframe to be visible | +| Assuming same-origin | Cross-origin has different timing | Always wait for iframe content explicitly | +| Ignoring nested iframes | Element not found | Chain frameLocator calls for nested frames | + +## Related References + +- **Locators**: See [locators.md](../core/locators.md) for selector strategies +- **Third-party services**: See [third-party.md](../advanced/third-party.md) for payment iframe patterns +- **Debugging**: See [debugging.md](../debugging/debugging.md) for troubleshooting iframe issues diff --git a/external/develop/devops/playwright-best-practices/browser-apis/service-workers.md b/external/develop/devops/playwright-best-practices/browser-apis/service-workers.md new file mode 100644 index 0000000..7603de3 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/browser-apis/service-workers.md @@ -0,0 +1,504 @@ +# Service Worker Testing + +## Table of Contents + +1. [Service Worker Basics](#service-worker-basics) +2. [Registration & Lifecycle](#registration--lifecycle) +3. [Cache Testing](#cache-testing) +4. [Offline Testing](#offline-testing) +5. [Push Notifications](#push-notifications) +6. [Background Sync](#background-sync) + +## Service Worker Basics + +### Waiting for Service Worker Registration + +```typescript +test("service worker registers", async ({ page }) => { + await page.goto("/pwa-app"); + + // Wait for SW to register + const swRegistered = await page.evaluate(async () => { + if (!("serviceWorker" in navigator)) return false; + + const registration = await navigator.serviceWorker.ready; + return !!registration.active; + }); + + expect(swRegistered).toBe(true); +}); +``` + +### Getting Service Worker State + +```typescript +test("check SW state", async ({ page }) => { + await page.goto("/"); + + const swState = await page.evaluate(async () => { + const registration = await navigator.serviceWorker.getRegistration(); + if (!registration) return null; + + return { + installing: !!registration.installing, + waiting: !!registration.waiting, + active: !!registration.active, + scope: registration.scope, + }; + }); + + expect(swState?.active).toBe(true); + expect(swState?.scope).toContain(page.url()); +}); +``` + +### Service Worker Context + +```typescript +test("access service worker", async ({ context, page }) => { + await page.goto("/pwa-app"); + + // Get all service workers in context + const workers = context.serviceWorkers(); + + // Wait for service worker if not yet available + if (workers.length === 0) { + await context.waitForEvent("serviceworker"); + } + + const sw = context.serviceWorkers()[0]; + expect(sw.url()).toContain("sw.js"); +}); +``` + +## Registration & Lifecycle + +### Testing SW Update Flow + +```typescript +test("service worker updates", async ({ page }) => { + await page.goto("/pwa-app"); + + // Check for update + const hasUpdate = await page.evaluate(async () => { + const registration = await navigator.serviceWorker.ready; + await registration.update(); + + return new Promise((resolve) => { + if (registration.waiting) { + resolve(true); + } else { + registration.addEventListener("updatefound", () => { + resolve(true); + }); + // Timeout if no update + setTimeout(() => resolve(false), 5000); + } + }); + }); + + // If update found, test skip waiting flow + if (hasUpdate) { + await page.evaluate(async () => { + const registration = await navigator.serviceWorker.ready; + registration.waiting?.postMessage({ type: "SKIP_WAITING" }); + }); + + // Wait for controller change + await page.evaluate(() => { + return new Promise((resolve) => { + navigator.serviceWorker.addEventListener("controllerchange", () => { + resolve(); + }); + }); + }); + } +}); +``` + +### Testing SW Installation + +```typescript +test("verify SW install event", async ({ context, page }) => { + // Listen for service worker before navigating + const swPromise = context.waitForEvent("serviceworker"); + + await page.goto("/pwa-app"); + + const sw = await swPromise; + + // Evaluate in SW context + const swVersion = await sw.evaluate(() => { + // Access SW globals + return (self as any).SW_VERSION || "unknown"; + }); + + expect(swVersion).toBe("1.0.0"); +}); +``` + +### Unregistering Service Workers + +```typescript +test.beforeEach(async ({ page }) => { + await page.goto("/"); + + // Unregister all service workers for clean state + await page.evaluate(async () => { + const registrations = await navigator.serviceWorker.getRegistrations(); + await Promise.all(registrations.map((r) => r.unregister())); + }); + + // Clear caches + await page.evaluate(async () => { + const cacheNames = await caches.keys(); + await Promise.all(cacheNames.map((name) => caches.delete(name))); + }); +}); +``` + +## Cache Testing + +### Verifying Cached Resources + +```typescript +test("assets are cached", async ({ page }) => { + await page.goto("/pwa-app"); + + // Wait for SW to cache assets + await page.evaluate(async () => { + await navigator.serviceWorker.ready; + }); + + // Check cache contents + const cachedUrls = await page.evaluate(async () => { + const cache = await caches.open("app-cache-v1"); + const requests = await cache.keys(); + return requests.map((r) => r.url); + }); + + expect(cachedUrls).toContain(expect.stringContaining("/styles.css")); + expect(cachedUrls).toContain(expect.stringContaining("/app.js")); +}); +``` + +### Testing Cache Strategies + +```typescript +test("cache-first strategy", async ({ page }) => { + await page.goto("/pwa-app"); + + // Wait for initial cache + await page.waitForFunction(async () => { + const cache = await caches.open("app-cache-v1"); + const keys = await cache.keys(); + return keys.length > 0; + }); + + // Block network for cached resources + await page.route("**/styles.css", (route) => route.abort()); + + // Reload - should work from cache + await page.reload(); + + // Verify page still styled (CSS loaded from cache) + const hasStyles = await page.evaluate(() => { + const body = document.body; + const styles = window.getComputedStyle(body); + return styles.fontFamily !== ""; // Has custom font from CSS + }); + + expect(hasStyles).toBe(true); +}); +``` + +### Testing Cache Updates + +```typescript +test("cache updates on new version", async ({ page }) => { + await page.goto("/pwa-app"); + + // Get initial cache + const initialCacheKeys = await page.evaluate(async () => { + const cache = await caches.open("app-cache-v1"); + const keys = await cache.keys(); + return keys.map((r) => r.url); + }); + + // Simulate app update by mocking SW response + await page.route("**/sw.js", (route) => { + route.fulfill({ + contentType: "application/javascript", + body: ` + const VERSION = 'v2'; + self.addEventListener('install', (e) => { + e.waitUntil(caches.open('app-cache-v2')); + self.skipWaiting(); + }); + `, + }); + }); + + // Trigger update + await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + await reg.update(); + }); + + // Verify new cache exists + await page.waitForFunction(async () => { + return await caches.has("app-cache-v2"); + }); +}); +``` + +## Offline Testing + +This section covers **offline-first apps (PWAs)** that are designed to work offline using service workers, caching, and background sync. For testing **unexpected network failures** (error recovery, graceful degradation), see [error-testing.md](error-testing.md#offline-testing). + +### Simulating Offline Mode + +```typescript +test("app works offline", async ({ page, context }) => { + await page.goto("/pwa-app"); + + // Ensure SW is active and content cached + await page.evaluate(async () => { + await navigator.serviceWorker.ready; + }); + await page.waitForTimeout(1000); // Allow caching to complete + + // Go offline + await context.setOffline(true); + + // Navigate to cached page + await page.reload(); + + // Verify content loads + await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible(); + + // Verify offline indicator + await expect(page.locator(".offline-badge")).toBeVisible(); + + // Go back online + await context.setOffline(false); + await expect(page.locator(".offline-badge")).not.toBeVisible(); +}); +``` + +### Testing Offline Fallback + +```typescript +test("shows offline page for uncached routes", async ({ page, context }) => { + await page.goto("/pwa-app"); + await page.evaluate(() => navigator.serviceWorker.ready); + + // Go offline + await context.setOffline(true); + + // Navigate to uncached page + await page.goto("/uncached-page"); + + // Should show offline fallback + await expect(page.getByText("You are offline")).toBeVisible(); + await expect(page.getByRole("button", { name: "Retry" })).toBeVisible(); +}); +``` + +### Testing Offline Form Submission + +```typescript +test("queues form submission offline", async ({ page, context }) => { + await page.goto("/pwa-app/form"); + + // Go offline + await context.setOffline(true); + + // Submit form + await page.getByLabel("Message").fill("Offline message"); + await page.getByRole("button", { name: "Send" }).click(); + + // Should show queued status + await expect(page.getByText("Queued for sync")).toBeVisible(); + + // Go online + await context.setOffline(false); + + // Trigger sync (or wait for automatic) + await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + // Manually trigger sync for testing + await (reg as any).sync?.register("form-sync"); + }); + + // Verify submission completed + await expect(page.getByText("Message sent")).toBeVisible({ timeout: 10000 }); +}); +``` + +## Push Notifications + +### Mocking Push Subscription + +```typescript +test("handles push subscription", async ({ page, context }) => { + // Grant notification permission + await context.grantPermissions(["notifications"]); + + await page.goto("/pwa-app"); + + // Subscribe to push + const subscription = await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + const sub = await reg.pushManager.subscribe({ + userVisibleOnly: true, + applicationServerKey: "test-key", + }); + return sub.toJSON(); + }); + + expect(subscription.endpoint).toBeDefined(); +}); +``` + +### Testing Push Message Handling + +```typescript +test("handles push notification", async ({ context, page }) => { + await context.grantPermissions(["notifications"]); + await page.goto("/pwa-app"); + + // Wait for SW + const swPromise = context.waitForEvent("serviceworker"); + const sw = await swPromise; + + // Simulate push message to service worker + await sw.evaluate(async () => { + // Dispatch push event + const pushEvent = new PushEvent("push", { + data: new PushMessageData( + JSON.stringify({ title: "Test", body: "Push message" }), + ), + }); + self.dispatchEvent(pushEvent); + }); + + // Note: Actual notification display testing is limited in Playwright + // Focus on verifying the SW handles the push correctly +}); +``` + +### Testing Notification Click + +```typescript +test("notification click opens page", async ({ context, page }) => { + await context.grantPermissions(["notifications"]); + await page.goto("/pwa-app"); + + // Store notification URL target + let notificationUrl = ""; + + // Listen for new pages (notification click opens new page) + context.on("page", (newPage) => { + notificationUrl = newPage.url(); + }); + + // Trigger notification via SW + await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + await reg.showNotification("Test", { + body: "Click me", + data: { url: "/notification-target" }, + }); + }); + + // Simulate clicking notification (via SW) + const sw = context.serviceWorkers()[0]; + await sw.evaluate(() => { + self.dispatchEvent( + new NotificationEvent("notificationclick", { + notification: { data: { url: "/notification-target" } } as any, + }), + ); + }); + + // Verify navigation occurred + await page.waitForTimeout(1000); + // Check if new page opened or current page navigated +}); +``` + +## Background Sync + +### Testing Background Sync Registration + +```typescript +test("registers background sync", async ({ page }) => { + await page.goto("/pwa-app"); + + // Register sync + const syncRegistered = await page.evaluate(async () => { + const reg = await navigator.serviceWorker.ready; + if (!("sync" in reg)) return false; + + await (reg as any).sync.register("my-sync"); + return true; + }); + + expect(syncRegistered).toBe(true); +}); +``` + +### Testing Sync Event + +```typescript +test("sync event fires when online", async ({ context, page }) => { + await page.goto("/pwa-app"); + + // Queue data while offline + await context.setOffline(true); + + await page.evaluate(async () => { + // Store data in IndexedDB for sync + const db = await openDB(); + await db.put("sync-queue", { id: 1, data: "test" }); + + // Register sync + const reg = await navigator.serviceWorker.ready; + await (reg as any).sync.register("data-sync"); + }); + + // Track sync completion + await page.evaluate(() => { + window.syncCompleted = false; + navigator.serviceWorker.addEventListener("message", (e) => { + if (e.data.type === "SYNC_COMPLETE") { + window.syncCompleted = true; + } + }); + }); + + // Go online + await context.setOffline(false); + + // Wait for sync to complete + await page.waitForFunction(() => window.syncCompleted, { timeout: 10000 }); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ------------------------------ | ----------------------- | -------------------------------------------- | +| Not clearing SW between tests | Tests affect each other | Unregister SW in beforeEach | +| Not waiting for SW ready | Race conditions | Always await `navigator.serviceWorker.ready` | +| Testing in isolation only | Misses real SW behavior | Test with actual caching | +| Hardcoded timeouts for caching | Flaky tests | Wait for cache to populate | +| Ignoring SW update cycle | Missing update bugs | Test install, activate, update flows | + +## Related References + +- **Network Failures**: See [error-testing.md](error-testing.md#offline-testing) for unexpected network failure patterns +- **Browser APIs**: See [browser-apis.md](browser-apis.md) for permissions +- **Network Mocking**: See [network-advanced.md](../advanced/network-advanced.md) for network interception +- **Browser Extensions**: See [browser-extensions.md](../testing-patterns/browser-extensions.md) for extension service worker patterns diff --git a/external/develop/devops/playwright-best-practices/browser-apis/websockets.md b/external/develop/devops/playwright-best-practices/browser-apis/websockets.md new file mode 100644 index 0000000..075a997 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/browser-apis/websockets.md @@ -0,0 +1,403 @@ +# WebSocket & Real-Time Testing + +## Table of Contents + +1. [WebSocket Basics](#websocket-basics) +2. [Mocking WebSocket Messages](#mocking-websocket-messages) +3. [Testing Real-Time Features](#testing-real-time-features) +4. [Server-Sent Events](#server-sent-events) +5. [Reconnection Testing](#reconnection-testing) + +## WebSocket Basics + +### Wait for WebSocket Connection + +```typescript +test("chat connects via websocket", async ({ page }) => { + // Listen for WebSocket connection + const wsPromise = page.waitForEvent("websocket"); + + await page.goto("/chat"); + + const ws = await wsPromise; + expect(ws.url()).toContain("/ws/chat"); + + // Wait for connection to be established + await ws.waitForEvent("framesent"); +}); +``` + +### Monitor WebSocket Messages + +```typescript +test("receives real-time updates", async ({ page }) => { + const messages: string[] = []; + + // Set up listener before navigation + page.on("websocket", (ws) => { + ws.on("framereceived", (frame) => { + messages.push(frame.payload as string); + }); + }); + + await page.goto("/dashboard"); + + // Wait for some messages + await expect.poll(() => messages.length).toBeGreaterThan(0); + + // Verify message format + const data = JSON.parse(messages[0]); + expect(data).toHaveProperty("type"); +}); +``` + +### Capture Sent Messages + +```typescript +test("sends correct message format", async ({ page }) => { + const sentMessages: string[] = []; + + page.on("websocket", (ws) => { + ws.on("framesent", (frame) => { + sentMessages.push(frame.payload as string); + }); + }); + + await page.goto("/chat"); + await page.getByLabel("Message").fill("Hello!"); + await page.getByRole("button", { name: "Send" }).click(); + + // Verify sent message + await expect.poll(() => sentMessages.length).toBeGreaterThan(0); + + const sent = JSON.parse(sentMessages[sentMessages.length - 1]); + expect(sent).toEqual({ + type: "message", + content: "Hello!", + }); +}); +``` + +## Mocking WebSocket Messages + +### Inject Messages via Page Evaluate + +```typescript +test("displays incoming chat message", async ({ page }) => { + await page.goto("/chat"); + + // Wait for WebSocket to be ready + await page.waitForFunction( + () => (window as any).chatSocket?.readyState === 1, + ); + + // Simulate incoming message + await page.evaluate(() => { + const event = new MessageEvent("message", { + data: JSON.stringify({ + type: "message", + from: "Alice", + content: "Hello there!", + }), + }); + (window as any).chatSocket.dispatchEvent(event); + }); + + await expect(page.getByText("Alice: Hello there!")).toBeVisible(); +}); +``` + +### Mock WebSocket with Route Handler + +```typescript +test("mock websocket entirely", async ({ page, context }) => { + // Intercept the WebSocket upgrade + await context.route("**/ws/**", async (route) => { + // For WebSocket routes, we can't fulfill directly + // Instead, use page.evaluate to mock the client-side + }); + + // Alternative: Mock at application level + await page.addInitScript(() => { + const OriginalWebSocket = window.WebSocket; + (window as any).WebSocket = function (url: string) { + const ws = { + readyState: 1, + send: (data: string) => { + console.log("WS Send:", data); + }, + close: () => {}, + addEventListener: () => {}, + removeEventListener: () => {}, + }; + setTimeout(() => ws.onopen?.(), 100); + return ws; + }; + }); + + await page.goto("/chat"); +}); +``` + +### WebSocket Mock Fixture + +```typescript +// fixtures/websocket.fixture.ts +import { test as base, Page } from "@playwright/test"; + +type WsMessage = { type: string; [key: string]: any }; + +type WebSocketFixtures = { + mockWebSocket: { + injectMessage: (message: WsMessage) => Promise; + getSentMessages: () => Promise; + }; +}; + +export const test = base.extend({ + mockWebSocket: async ({ page }, use) => { + const sentMessages: WsMessage[] = []; + + // Capture sent messages + await page.addInitScript(() => { + (window as any).__wsSent = []; + const OriginalWebSocket = window.WebSocket; + window.WebSocket = function (url: string) { + const ws = new OriginalWebSocket(url); + const originalSend = ws.send.bind(ws); + ws.send = (data: string) => { + (window as any).__wsSent.push(JSON.parse(data)); + originalSend(data); + }; + (window as any).__ws = ws; + return ws; + } as any; + }); + + await use({ + injectMessage: async (message) => { + await page.evaluate((msg) => { + const event = new MessageEvent("message", { + data: JSON.stringify(msg), + }); + (window as any).__ws?.dispatchEvent(event); + }, message); + }, + getSentMessages: async () => { + return page.evaluate(() => (window as any).__wsSent || []); + }, + }); + }, +}); + +// Usage +test("chat with mocked websocket", async ({ page, mockWebSocket }) => { + await page.goto("/chat"); + + // Inject incoming message + await mockWebSocket.injectMessage({ + type: "message", + from: "Bob", + content: "Hi!", + }); + + await expect(page.getByText("Bob: Hi!")).toBeVisible(); + + // Send a reply + await page.getByLabel("Message").fill("Hello Bob!"); + await page.getByRole("button", { name: "Send" }).click(); + + // Verify sent message + const sent = await mockWebSocket.getSentMessages(); + expect(sent).toContainEqual( + expect.objectContaining({ content: "Hello Bob!" }), + ); +}); +``` + +## Testing Real-Time Features + +### Live Notifications + +```typescript +test("displays live notification", async ({ page }) => { + await page.goto("/dashboard"); + + // Simulate notification via WebSocket + await page.evaluate(() => { + const event = new MessageEvent("message", { + data: JSON.stringify({ + type: "notification", + title: "New Order", + message: "Order #123 received", + }), + }); + (window as any).notificationSocket.dispatchEvent(event); + }); + + await expect(page.getByRole("alert")).toContainText("Order #123 received"); +}); +``` + +### Live Data Updates + +```typescript +test("updates stock price in real-time", async ({ page }) => { + await page.goto("/stocks/AAPL"); + + const priceElement = page.getByTestId("stock-price"); + const initialPrice = await priceElement.textContent(); + + // Simulate price update + await page.evaluate(() => { + const event = new MessageEvent("message", { + data: JSON.stringify({ + type: "price_update", + symbol: "AAPL", + price: 150.25, + }), + }); + (window as any).stockSocket.dispatchEvent(event); + }); + + await expect(priceElement).not.toHaveText(initialPrice!); + await expect(priceElement).toContainText("150.25"); +}); +``` + +### Collaborative Editing + +```typescript +test("shows collaborator cursor", async ({ page }) => { + await page.goto("/document/123"); + + // Simulate another user's cursor position + await page.evaluate(() => { + const event = new MessageEvent("message", { + data: JSON.stringify({ + type: "cursor", + userId: "user-456", + userName: "Alice", + position: { x: 100, y: 200 }, + }), + }); + (window as any).docSocket.dispatchEvent(event); + }); + + await expect(page.getByTestId("cursor-user-456")).toBeVisible(); + await expect(page.getByText("Alice")).toBeVisible(); +}); +``` + +## Server-Sent Events + +### Test SSE Updates + +```typescript +test("receives SSE updates", async ({ page }) => { + // Mock SSE endpoint + await page.route("**/api/events", (route) => { + route.fulfill({ + status: 200, + headers: { + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + Connection: "keep-alive", + }, + body: `data: {"type":"update","value":42}\n\n`, + }); + }); + + await page.goto("/live-data"); + + await expect(page.getByTestId("value")).toHaveText("42"); +}); +``` + +### Simulate Multiple SSE Events + +```typescript +test("handles multiple SSE events", async ({ page }) => { + await page.route("**/api/events", async (route) => { + const encoder = new TextEncoder(); + const events = [ + `data: {"count":1}\n\n`, + `data: {"count":2}\n\n`, + `data: {"count":3}\n\n`, + ]; + + route.fulfill({ + status: 200, + headers: { "Content-Type": "text/event-stream" }, + body: events.join(""), + }); + }); + + await page.goto("/counter"); + + // Should receive all events + await expect(page.getByTestId("count")).toHaveText("3"); +}); +``` + +## Reconnection Testing + +### Test Connection Loss + +```typescript +test("handles connection loss gracefully", async ({ page }) => { + await page.goto("/chat"); + + // Simulate connection close + await page.evaluate(() => { + (window as any).chatSocket.close(); + }); + + // Should show disconnected state + await expect(page.getByText("Reconnecting...")).toBeVisible(); +}); +``` + +### Test Reconnection + +```typescript +test("reconnects after connection loss", async ({ page }) => { + await page.goto("/chat"); + + // Simulate disconnect + await page.evaluate(() => { + (window as any).chatSocket.close(); + }); + + await expect(page.getByText("Reconnecting...")).toBeVisible(); + + // Simulate reconnection + await page.evaluate(() => { + const event = new Event("open"); + (window as any).chatSocket = { readyState: 1 }; + (window as any).chatSocket.dispatchEvent?.(event); + }); + + // Force component to re-check connection + await page.evaluate(() => { + window.dispatchEvent(new Event("online")); + }); + + await expect(page.getByText("Connected")).toBeVisible(); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ------------------------------------- | ----------------------------- | ---------------------------------- | +| Not waiting for WebSocket ready | Messages sent too early | Wait for `readyState === 1` | +| Testing against real WebSocket server | Flaky, timing-dependent | Mock WebSocket messages | +| Ignoring connection state | Tests pass but feature broken | Test connected/disconnected states | +| No cleanup of listeners | Memory leaks in tests | Clean up event listeners | + +## Related References + +- **Network**: See [network-advanced.md](../advanced/network-advanced.md) for HTTP mocking patterns +- **Assertions**: See [assertions-waiting.md](../core/assertions-waiting.md) for polling patterns +- **Multi-User**: See [multi-user.md](../advanced/multi-user.md) for real-time collaboration testing with multiple users diff --git a/external/develop/devops/playwright-best-practices/core/annotations.md b/external/develop/devops/playwright-best-practices/core/annotations.md new file mode 100644 index 0000000..ac0f890 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/core/annotations.md @@ -0,0 +1,424 @@ +# Test Annotations & Organization + +## Table of Contents + +1. [Skip Annotations](#skip-annotations) +2. [Fixme & Fail Annotations](#fixme--fail-annotations) +3. [Slow Tests](#slow-tests) +4. [Test Steps](#test-steps) +5. [Custom Annotations](#custom-annotations) +6. [Conditional Annotations](#conditional-annotations) + +## Skip Annotations + +### Basic Skip + +```typescript +// Skip unconditionally +test.skip("feature not implemented", async ({ page }) => { + // This test won't run +}); + +// Skip with reason +test("payment flow", async ({ page }) => { + test.skip(true, "Payment gateway in maintenance"); + // Test body won't execute +}); +``` + +### Conditional Skip + +```typescript +test("webkit-specific feature", async ({ page, browserName }) => { + test.skip(browserName !== "webkit", "This feature only works in WebKit"); + + await page.goto("/webkit-feature"); +}); + +test("production only", async ({ page }) => { + test.skip(process.env.ENV !== "production", "Only runs against production"); + + await page.goto("/prod-feature"); +}); +``` + +### Skip by Platform + +```typescript +test("windows-specific", async ({ page }) => { + test.skip(process.platform !== "win32", "Windows only"); +}); + +test("not on CI", async ({ page }) => { + test.skip(!!process.env.CI, "Skipped in CI environment"); +}); +``` + +### Skip Describe Block + +```typescript +test.describe("Admin features", () => { + test.skip( + ({ browserName }) => browserName === "firefox", + "Firefox admin bug", + ); + + test("admin dashboard", async ({ page }) => { + // Skipped in Firefox + }); + + test("admin settings", async ({ page }) => { + // Skipped in Firefox + }); +}); +``` + +## Fixme & Fail Annotations + +### Fixme - Known Issues + +```typescript +// Mark test as needing fix (skips the test) +test.fixme("broken after refactor", async ({ page }) => { + // Test won't run but is tracked +}); + +// Conditional fixme +test("flaky on CI", async ({ page }) => { + test.fixme(!!process.env.CI, "Investigate CI flakiness - ticket #123"); + + await page.goto("/flaky-feature"); +}); +``` + +### Fail - Expected Failures + +```typescript +// Test is expected to fail (runs but expects failure) +test("known bug", async ({ page }) => { + test.fail(); + + await page.goto("/buggy-page"); + // If this passes, the test fails (bug was fixed!) + await expect(page.getByText("Working")).toBeVisible(); +}); + +// Conditional fail +test("fails on webkit", async ({ page, browserName }) => { + test.fail(browserName === "webkit", "WebKit rendering bug #456"); + + await page.goto("/render-test"); + await expect(page.getByTestId("element")).toHaveCSS("width", "100px"); +}); +``` + +### Difference Between Skip, Fixme, Fail + +| Annotation | Runs? | Use Case | +| -------------- | ----- | -------------------------------- | +| `test.skip()` | No | Feature not applicable | +| `test.fixme()` | No | Known bug, needs investigation | +| `test.fail()` | Yes | Expected to fail, tracking a bug | + +## Slow Tests + +### Mark Slow Tests + +```typescript +// Triple the default timeout +test("large data import", async ({ page }) => { + test.slow(); + + await page.goto("/import"); + await page.setInputFiles("#file", "large-file.csv"); + await page.getByRole("button", { name: "Import" }).click(); + + await expect(page.getByText("Import complete")).toBeVisible(); +}); + +// Conditional slow +test("video processing", async ({ page, browserName }) => { + test.slow(browserName === "webkit", "WebKit video processing is slow"); + + await page.goto("/video-editor"); +}); +``` + +### Custom Timeout + +```typescript +test("very long operation", async ({ page }) => { + // Set specific timeout (in milliseconds) + test.setTimeout(120000); // 2 minutes + + await page.goto("/long-operation"); +}); + +// Timeout for describe block +test.describe("Integration tests", () => { + test.describe.configure({ timeout: 60000 }); + + test("test 1", async ({ page }) => { + // Has 60 second timeout + }); +}); +``` + +## Test Steps + +### Basic Steps + +```typescript +test("checkout flow", async ({ page }) => { + await test.step("Add item to cart", async () => { + await page.goto("/products"); + await page.getByRole("button", { name: "Add to Cart" }).click(); + }); + + await test.step("Go to checkout", async () => { + await page.getByRole("link", { name: "Cart" }).click(); + await page.getByRole("button", { name: "Checkout" }).click(); + }); + + await test.step("Fill shipping info", async () => { + await page.getByLabel("Address").fill("123 Test St"); + await page.getByLabel("City").fill("Test City"); + }); + + await test.step("Complete payment", async () => { + await page.getByLabel("Card").fill("4242424242424242"); + await page.getByRole("button", { name: "Pay" }).click(); + }); + + await expect(page.getByText("Order confirmed")).toBeVisible(); +}); +``` + +### Nested Steps + +```typescript +test("user registration", async ({ page }) => { + await test.step("Fill registration form", async () => { + await page.goto("/register"); + + await test.step("Personal info", async () => { + await page.getByLabel("Name").fill("John Doe"); + await page.getByLabel("Email").fill("john@example.com"); + }); + + await test.step("Security", async () => { + await page.getByLabel("Password").fill("SecurePass123"); + await page.getByLabel("Confirm Password").fill("SecurePass123"); + }); + }); + + await test.step("Submit and verify", async () => { + await page.getByRole("button", { name: "Register" }).click(); + await expect(page.getByText("Welcome")).toBeVisible(); + }); +}); +``` + +### Steps with Return Values + +```typescript +test("verify order", async ({ page }) => { + const orderId = await test.step("Create order", async () => { + await page.goto("/checkout"); + await page.getByRole("button", { name: "Place Order" }).click(); + + // Return value from step + return await page.getByTestId("order-id").textContent(); + }); + + await test.step("Verify order details", async () => { + await page.goto(`/orders/${orderId}`); + await expect(page.getByText(`Order #${orderId}`)).toBeVisible(); + }); +}); +``` + +### Step in Page Object + +```typescript +// pages/checkout.page.ts +export class CheckoutPage { + async fillShippingInfo(address: string, city: string) { + await test.step("Fill shipping information", async () => { + await this.page.getByLabel("Address").fill(address); + await this.page.getByLabel("City").fill(city); + }); + } + + async completePayment(cardNumber: string) { + await test.step("Complete payment", async () => { + await this.page.getByLabel("Card").fill(cardNumber); + await this.page.getByRole("button", { name: "Pay" }).click(); + }); + } +} +``` + +## Custom Annotations + +### Add Annotations + +```typescript +test("important feature", async ({ page }, testInfo) => { + // Add custom annotation + testInfo.annotations.push({ + type: "priority", + description: "high", + }); + + testInfo.annotations.push({ + type: "ticket", + description: "JIRA-123", + }); + + await page.goto("/feature"); +}); +``` + +### Annotation Fixture + +```typescript +// fixtures/annotations.fixture.ts +import { test as base, TestInfo } from "@playwright/test"; + +type AnnotationFixtures = { + annotate: { + ticket: (id: string) => void; + priority: (level: "low" | "medium" | "high") => void; + owner: (name: string) => void; + }; +}; + +export const test = base.extend({ + annotate: async ({}, use, testInfo) => { + await use({ + ticket: (id) => { + testInfo.annotations.push({ type: "ticket", description: id }); + }, + priority: (level) => { + testInfo.annotations.push({ type: "priority", description: level }); + }, + owner: (name) => { + testInfo.annotations.push({ type: "owner", description: name }); + }, + }); + }, +}); + +// Usage +test("critical feature", async ({ page, annotate }) => { + annotate.ticket("JIRA-456"); + annotate.priority("high"); + annotate.owner("Alice"); + + await page.goto("/critical"); +}); +``` + +### Read Annotations in Reporter + +```typescript +// reporters/annotation-reporter.ts +import { Reporter, TestCase, TestResult } from "@playwright/test/reporter"; + +class AnnotationReporter implements Reporter { + onTestEnd(test: TestCase, result: TestResult) { + const ticket = test.annotations.find((a) => a.type === "ticket"); + const priority = test.annotations.find((a) => a.type === "priority"); + + if (ticket) { + console.log(`Test linked to: ${ticket.description}`); + } + + if (priority?.description === "high" && result.status === "failed") { + console.log(`HIGH PRIORITY FAILURE: ${test.title}`); + } + } +} + +export default AnnotationReporter; +``` + +## Conditional Annotations + +### Annotation Helper + +```typescript +// helpers/test-annotations.ts +import { test } from "@playwright/test"; + +export function skipInCI(reason = "Skipped in CI") { + test.skip(!!process.env.CI, reason); +} + +export function skipInBrowser(browser: string, reason: string) { + test.beforeEach(({ browserName }) => { + test.skip(browserName === browser, reason); + }); +} + +export function onlyInEnv(env: string) { + test.skip(process.env.ENV !== env, `Only runs in ${env}`); +} +``` + +```typescript +// tests/feature.spec.ts +import { skipInCI, onlyInEnv } from "../helpers/test-annotations"; + +test("local only feature", async ({ page }) => { + skipInCI("Uses local resources"); + + await page.goto("/local-feature"); +}); + +test("production check", async ({ page }) => { + onlyInEnv("production"); + + await page.goto("/prod-only"); +}); +``` + +### Describe-Level Conditions + +```typescript +test.describe("Mobile features", () => { + test.beforeEach(({ isMobile }) => { + test.skip(!isMobile, "Mobile only tests"); + }); + + test("touch gestures", async ({ page }) => { + // Only runs on mobile + }); +}); + +test.describe("Desktop features", () => { + test.beforeEach(({ isMobile }) => { + test.skip(isMobile, "Desktop only tests"); + }); + + test("hover interactions", async ({ page }) => { + // Only runs on desktop + }); +}); +``` + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| --------------------------- | ---------------------- | -------------------------------- | +| Skipping without reason | Hard to track why | Always provide description | +| Too many skipped tests | Test debt accumulates | Review and clean up regularly | +| Using skip instead of fixme | Loses intent | Use fixme for bugs, skip for N/A | +| Not using steps | Hard to debug failures | Group logical actions in steps | + +## Related References + +- **Test Tags**: See [test-tags.md](test-tags.md) for tagging and filtering tests with `--grep` +- **Test Organization**: See [test-suite-structure.md](test-suite-structure.md) for structuring tests +- **Debugging**: See [debugging.md](../debugging/debugging.md) for troubleshooting diff --git a/external/develop/devops/playwright-best-practices/core/assertions-waiting.md b/external/develop/devops/playwright-best-practices/core/assertions-waiting.md new file mode 100644 index 0000000..bd03dd8 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/core/assertions-waiting.md @@ -0,0 +1,361 @@ +# Assertions & Waiting + +## Table of Contents + +1. [Web-First Assertions](#web-first-assertions) +2. [Generic Assertions](#generic-assertions) +3. [Soft Assertions](#soft-assertions) +4. [Waiting Strategies](#waiting-strategies) +5. [Polling & Retrying](#polling--retrying) +6. [Custom Matchers](#custom-matchers) + +## Web-First Assertions + +Auto-retry until condition is met or timeout. Always prefer these over generic assertions. + +### Locator Assertions + +```typescript +import { expect } from "@playwright/test"; + +// Visibility +await expect(page.getByRole("button")).toBeVisible(); +await expect(page.getByRole("button")).toBeHidden(); +await expect(page.getByRole("button")).not.toBeVisible(); + +// Enabled/Disabled +await expect(page.getByRole("button")).toBeEnabled(); +await expect(page.getByRole("button")).toBeDisabled(); + +// Text content +await expect(page.getByRole("heading")).toHaveText("Welcome"); +await expect(page.getByRole("heading")).toHaveText(/welcome/i); +await expect(page.getByRole("heading")).toContainText("Welcome"); + +// Count +await expect(page.getByRole("listitem")).toHaveCount(5); + +// Attributes +await expect(page.getByRole("link")).toHaveAttribute("href", "/home"); +await expect(page.getByRole("img")).toHaveAttribute("alt", /logo/i); + +// CSS +await expect(page.getByRole("button")).toHaveClass(/primary/); +await expect(page.getByRole("button")).toHaveCSS("color", "rgb(0, 0, 255)"); + +// Input values +await expect(page.getByLabel("Email")).toHaveValue("user@example.com"); +await expect(page.getByLabel("Email")).toBeEmpty(); + +// Focus +await expect(page.getByLabel("Email")).toBeFocused(); + +// Checked state +await expect(page.getByRole("checkbox")).toBeChecked(); +await expect(page.getByRole("checkbox")).not.toBeChecked(); + +// Editable state +await expect(page.getByLabel("Name")).toBeEditable(); +``` + +### Page Assertions + +```typescript +// URL +await expect(page).toHaveURL("/dashboard"); +await expect(page).toHaveURL(/\/dashboard/); + +// Title +await expect(page).toHaveTitle("Dashboard - MyApp"); +await expect(page).toHaveTitle(/dashboard/i); +``` + +### Response Assertions + +```typescript +const response = await page.request.get("/api/users"); +await expect(response).toBeOK(); +await expect(response).not.toBeOK(); +``` + +## Generic Assertions + +Use for non-UI values. Do NOT retry - execute immediately. + +```typescript +// Equality +expect(value).toBe(5); +expect(object).toEqual({ name: "Test" }); +expect(array).toContain("item"); + +// Truthiness +expect(value).toBeTruthy(); +expect(value).toBeFalsy(); +expect(value).toBeNull(); +expect(value).toBeUndefined(); +expect(value).toBeDefined(); + +// Numbers +expect(value).toBeGreaterThan(5); +expect(value).toBeLessThanOrEqual(10); +expect(value).toBeCloseTo(5.5, 1); + +// Strings +expect(string).toMatch(/pattern/); +expect(string).toContain("substring"); + +// Arrays/Objects +expect(array).toHaveLength(3); +expect(object).toHaveProperty("key", "value"); + +// Exceptions +expect(() => fn()).toThrow(); +expect(() => fn()).toThrow("error message"); +await expect(asyncFn()).rejects.toThrow(); +``` + +## Soft Assertions + +Continue test execution after failure, report all failures at end. + +```typescript +test("check multiple elements", async ({ page }) => { + await page.goto("/dashboard"); + + // Won't stop on first failure + await expect.soft(page.getByRole("heading")).toHaveText("Dashboard"); + await expect.soft(page.getByRole("button", { name: "Save" })).toBeEnabled(); + await expect.soft(page.getByText("Welcome")).toBeVisible(); + + // Test continues; all failures reported at end +}); +``` + +### Soft Assertions with Early Exit + +```typescript +test("check form", async ({ page }) => { + await expect.soft(page.getByRole("form")).toBeVisible(); + + // Exit early if form not visible (pointless to check fields) + if (expect.soft.hasFailures()) { + return; + } + + await expect.soft(page.getByLabel("Name")).toBeVisible(); + await expect.soft(page.getByLabel("Email")).toBeVisible(); +}); +``` + +## Waiting Strategies + +### Auto-Waiting (Default) + +Actions automatically wait for: + +- Element to be attached to DOM +- Element to be visible +- Element to be stable (no animations) +- Element to be enabled +- Element to receive events + +```typescript +// These auto-wait +await page.click("button"); +await page.fill("input", "text"); +await page.getByRole("button").click(); +``` + +### Wait for Navigation + +```typescript +// Wait for URL change +await page.waitForURL("/dashboard"); +await page.waitForURL(/\/dashboard/); + +// Wait for navigation after action +await Promise.all([ + page.waitForURL("**/dashboard"), + page.click('a[href="/dashboard"]'), +]); + +// Or without Promise.all +const urlPromise = page.waitForURL("**/dashboard"); +await page.click("a"); +await urlPromise; +``` + +### Wait for Network + +```typescript +// Wait for specific response +const responsePromise = page.waitForResponse("**/api/users"); +await page.click("button"); +const response = await responsePromise; +expect(response.status()).toBe(200); + +// Wait for request +const requestPromise = page.waitForRequest("**/api/submit"); +await page.click("button"); +const request = await requestPromise; + +// Wait for no network activity +await page.waitForLoadState("networkidle"); +``` + +### Wait for Element State + +```typescript +// Wait for element to appear +await page.getByRole("dialog").waitFor({ state: "visible" }); + +// Wait for element to disappear +await page.getByText("Loading...").waitFor({ state: "hidden" }); + +// Wait for element to be attached +await page.getByTestId("result").waitFor({ state: "attached" }); + +// Wait for element to be detached +await page.getByTestId("modal").waitFor({ state: "detached" }); +``` + +### Wait for Function + +```typescript +// Wait for arbitrary condition +await page.waitForFunction(() => { + return document.querySelector(".loaded") !== null; +}); + +// With arguments +await page.waitForFunction( + (selector) => document.querySelector(selector)?.textContent === "Ready", + ".status", +); +``` + +## Polling & Retrying + +### toPass() for Polling + +Retry until block passes or times out: + +```typescript +await expect(async () => { + const response = await page.request.get("/api/status"); + expect(response.status()).toBe(200); + + const data = await response.json(); + expect(data.ready).toBe(true); +}).toPass({ + intervals: [1000, 2000, 5000], // Retry intervals + timeout: 30000, +}); +``` + +### expect.poll() + +Poll a function until assertion passes: + +```typescript +// Poll API until condition met +await expect + .poll( + async () => { + const response = await page.request.get("/api/job/123"); + return (await response.json()).status; + }, + { + intervals: [1000, 2000, 5000], + timeout: 30000, + }, + ) + .toBe("completed"); + +// Poll DOM value +await expect.poll(() => page.getByTestId("counter").textContent()).toBe("10"); +``` + +## Custom Matchers + +```typescript +// playwright.config.ts or fixtures +import { expect } from "@playwright/test"; + +expect.extend({ + async toHaveDataLoaded(page: Page) { + const locator = page.getByTestId("data-container"); + let pass = false; + let message = ""; + + try { + await expect(locator).toBeVisible(); + await expect(locator).not.toContainText("Loading"); + pass = true; + } catch (e) { + message = `Expected data to be loaded but found loading state`; + } + + return { pass, message: () => message }; + }, +}); + +// Extend TypeScript types +declare global { + namespace PlaywrightTest { + interface Matchers { + toHaveDataLoaded(): Promise; + } + } +} + +// Usage +await expect(page).toHaveDataLoaded(); +``` + +## Timeouts + +### Configure Timeouts + +```typescript +// playwright.config.ts +export default defineConfig({ + timeout: 30000, // Test timeout + expect: { + timeout: 5000, // Assertion timeout + }, +}); + +// Per-test timeout +test("long test", async ({ page }) => { + test.setTimeout(60000); + // ... +}); + +// Per-assertion timeout +await expect(page.getByRole("button")).toBeVisible({ timeout: 10000 }); +``` + +## Best Practices + +| Do | Don't | +| ------------------------------ | ------------------------------ | +| Use web-first assertions | Use generic assertions for DOM | +| Let auto-waiting work | Add unnecessary explicit waits | +| Use `toPass()` for polling | Write manual retry loops | +| Configure appropriate timeouts | Use `waitForTimeout()` | +| Check specific conditions | Wait for arbitrary time | + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| --------------------------------------------------------- | ----------------------------- | -------------------------------------------- | +| `await page.waitForTimeout(5000)` | Slow, flaky, arbitrary timing | Use auto-waiting or `waitForResponse` | +| `await new Promise(resolve => setTimeout(resolve, 1000))` | Same as above | Use `waitForResponse` or element state waits | +| Generic assertions on DOM elements | No auto-retry, flaky | Use web-first assertions with `expect()` | + +## Related References + +- **Debugging timeout issues**: See [debugging.md](../debugging/debugging.md) for troubleshooting +- **Fixing flaky tests**: See [debugging.md](../debugging/debugging.md) for race condition solutions +- **Network interception**: See [test-suite-structure.md](test-suite-structure.md) for API mocking diff --git a/external/develop/devops/playwright-best-practices/core/configuration.md b/external/develop/devops/playwright-best-practices/core/configuration.md new file mode 100644 index 0000000..66b9d33 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/core/configuration.md @@ -0,0 +1,452 @@ +# Playwright Configuration + +## Table of Contents + +1. [CLI Quick Reference](#cli-quick-reference) +2. [Decision Guide](#decision-guide) +3. [Production-Ready Config](#production-ready-config) +4. [Patterns](#patterns) +5. [Anti-Patterns](#anti-patterns) +6. [Troubleshooting](#troubleshooting) +7. [Related](#related) + +> **When to use**: Setting up a new project, adjusting timeouts, adding browser targets, configuring CI behavior, or managing environment-specific settings. + +## CLI Quick Reference + +```bash +npx playwright init # scaffold config + first test +npx playwright test --config=custom.config.ts # use alternate config +npx playwright test --project=chromium # run single project +npx playwright test --reporter=html # override reporter +npx playwright test --grep @smoke # run tests tagged @smoke +npx playwright test --grep-invert @slow # exclude @slow tests +npx playwright show-report # open last HTML report +DEBUG=pw:api npx playwright test # verbose logging +``` + +## Decision Guide + +### Timeout Selection + +| Symptom | Setting | Default | Recommended | +|---------|---------|---------|-------------| +| Test takes too long overall | `timeout` | 30s | 30-60s (max 120s) | +| Assertion retries too long/short | `expect.timeout` | 5s | 5-10s | +| `page.goto()` or `waitForURL()` times out | `navigationTimeout` | 30s | 10-30s | +| `click()`, `fill()` time out | `actionTimeout` | 0 (unlimited) | 10-15s | +| Dev server slow to start | `webServer.timeout` | 60s | 60-180s | + +### Server Management + +| Scenario | Approach | +|----------|----------| +| App in same repo | `webServer` with `reuseExistingServer: !process.env.CI` | +| Separate repos | Manual start or Docker Compose | +| Testing deployed environment | No `webServer`; set `baseURL` via env | +| Multiple services | Array of `webServer` entries | + +### Single vs Multi-Project + +| Scenario | Approach | +|----------|----------| +| Early development | Single project (chromium only) | +| Pre-release validation | Multi-project: chromium + firefox + webkit | +| Mobile-responsive app | Add mobile projects alongside desktop | +| Auth + non-auth tests | Setup project with dependencies | +| Tight CI budget | Chromium on PRs; all browsers on main | + +### globalSetup vs Setup Projects vs Fixtures + +| Need | Use | +|------|-----| +| One-time DB seed | `globalSetup` | +| Shared browser auth | Setup project with `dependencies` | +| Per-test isolated state | Custom fixture via `test.extend()` | +| Cleanup after all tests | `globalTeardown` | + +## Production-Ready Config + +```ts +// playwright.config.ts +import { defineConfig, devices } from '@playwright/test'; +import dotenv from 'dotenv'; +import path from 'path'; + +dotenv.config({ path: path.resolve(__dirname, '.env') }); + +export default defineConfig({ + testDir: './e2e', + testMatch: '**/*.spec.ts', + + fullyParallel: true, + forbidOnly: !!process.env.CI, + retries: process.env.CI ? 2 : 0, + workers: process.env.CI ? '50%' : undefined, + + reporter: process.env.CI + ? [['html', { open: 'never' }], ['github']] + : [['html', { open: 'on-failure' }]], + + timeout: 30_000, + expect: { timeout: 5_000 }, + + use: { + baseURL: process.env.BASE_URL || 'http://localhost:4000', + actionTimeout: 10_000, + navigationTimeout: 15_000, + trace: 'on-first-retry', + screenshot: 'only-on-failure', + video: 'retain-on-failure', + locale: 'en-US', + timezoneId: 'America/Los_Angeles', + }, + + projects: [ + { name: 'chromium', use: { ...devices['Desktop Chrome'] } }, + { name: 'firefox', use: { ...devices['Desktop Firefox'] } }, + { name: 'webkit', use: { ...devices['Desktop Safari'] } }, + { name: 'mobile-chrome', use: { ...devices['Pixel 7'] } }, + { name: 'mobile-safari', use: { ...devices['iPhone 14'] } }, + ], + + webServer: { + command: 'npm run start', + url: 'http://localhost:4000', + reuseExistingServer: !process.env.CI, + timeout: 120_000, + stdout: 'pipe', + stderr: 'pipe', + }, +}); +``` + +## Patterns + +### Environment-Specific Configuration + +**Use when**: Tests run against dev, staging, and production environments. + +```ts +// playwright.config.ts +import { defineConfig } from '@playwright/test'; +import dotenv from 'dotenv'; +import path from 'path'; + +const ENV = process.env.TEST_ENV || 'local'; +dotenv.config({ path: path.resolve(__dirname, `.env.${ENV}`) }); + +const envConfig: Record = { + local: { baseURL: 'http://localhost:4000', retries: 0 }, + staging: { baseURL: 'https://staging.myapp.com', retries: 2 }, + prod: { baseURL: 'https://myapp.com', retries: 2 }, +}; + +export default defineConfig({ + testDir: './e2e', + retries: envConfig[ENV].retries, + use: { baseURL: envConfig[ENV].baseURL }, +}); +``` + +```bash +TEST_ENV=staging npx playwright test +TEST_ENV=prod npx playwright test --grep @smoke +``` + +### Setup Project with Dependencies + +**Use when**: Tests need shared authentication state before running. + +```ts +// playwright.config.ts +import { defineConfig, devices } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + projects: [ + { + name: 'setup', + testMatch: /auth\.setup\.ts/, + }, + { + name: 'chromium', + use: { + ...devices['Desktop Chrome'], + storageState: 'playwright/.auth/session.json', + }, + dependencies: ['setup'], + }, + { + name: 'firefox', + use: { + ...devices['Desktop Firefox'], + storageState: 'playwright/.auth/session.json', + }, + dependencies: ['setup'], + }, + ], +}); +``` + +```ts +// e2e/auth.setup.ts +import { test as setup, expect } from '@playwright/test'; + +const authFile = 'playwright/.auth/session.json'; + +setup('authenticate', async ({ page }) => { + await page.goto('/login'); + await page.getByLabel('Username').fill('testuser@example.com'); + await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!); + await page.getByRole('button', { name: 'Log in' }).click(); + await expect(page.getByRole('heading', { name: 'Home' })).toBeVisible(); + await page.context().storageState({ path: authFile }); +}); +``` + +### webServer with Build Step + +**Use when**: Tests need a running application server managed by Playwright. + +```ts +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + use: { baseURL: 'http://localhost:4000' }, + webServer: { + command: process.env.CI + ? 'npm run build && npm run preview' + : 'npm run dev', + url: 'http://localhost:4000', + reuseExistingServer: !process.env.CI, + timeout: 120_000, + env: { + NODE_ENV: 'test', + DB_URL: process.env.DB_URL || 'postgresql://localhost:5432/testdb', + }, + }, +}); +``` + +### globalSetup / globalTeardown + +**Use when**: One-time non-browser work like seeding a database. Runs once per test run. + +```ts +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + globalSetup: './e2e/setup.ts', + globalTeardown: './e2e/teardown.ts', +}); +``` + +```ts +// e2e/setup.ts +import { FullConfig } from '@playwright/test'; + +export default async function globalSetup(config: FullConfig) { + const { execSync } = await import('child_process'); + execSync('npx prisma db seed', { stdio: 'inherit' }); + process.env.TEST_RUN_ID = `run-${Date.now()}`; +} +``` + +```ts +// e2e/teardown.ts +import { FullConfig } from '@playwright/test'; + +export default async function globalTeardown(config: FullConfig) { + const { execSync } = await import('child_process'); + execSync('npx prisma db push --force-reset', { stdio: 'inherit' }); +} +``` + +### Environment Variables with .env + +**Use when**: Managing secrets, URLs, or feature flags without hardcoding. + +```bash +# .env.example (commit this) +BASE_URL=http://localhost:4000 +TEST_PASSWORD= +API_KEY= + +# .env.local (gitignored) +BASE_URL=http://localhost:4000 +TEST_PASSWORD=secret123 +API_KEY=dev-key-abc + +# .env.staging (gitignored) +BASE_URL=https://staging.myapp.com +TEST_PASSWORD=staging-pass +API_KEY=staging-key-xyz +``` + +```bash +# .gitignore +.env +.env.local +.env.staging +.env.production +playwright/.auth/ +``` + +Install dotenv: + +```bash +npm install -D dotenv +``` + +### Tag-Based Test Filtering + +**Use when**: Running subsets of tests in different CI stages (PR vs nightly). + +```ts +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + + // Filter by tags in CI + grep: process.env.CI ? /@smoke|@critical/ : undefined, + grepInvert: process.env.CI ? /@flaky/ : undefined, +}); +``` + +**Project-specific filtering:** + +```ts +// playwright.config.ts +import { defineConfig, devices } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + projects: [ + { + name: 'smoke', + grep: /@smoke/, + use: { ...devices['Desktop Chrome'] }, + }, + { + name: 'regression', + grepInvert: /@smoke/, + use: { ...devices['Desktop Chrome'] }, + }, + { + name: 'critical-only', + grep: /@critical/, + use: { ...devices['Desktop Chrome'] }, + }, + ], +}); +``` + +```bash +# Run specific project +npx playwright test --project=smoke +npx playwright test --project=regression +``` + +### Artifact Collection Strategy + +| Setting | Local | CI | Reason | +|---------|-------|-----|--------| +| `trace` | `'off'` | `'on-first-retry'` | Traces are large; collect on failure only | +| `screenshot` | `'off'` | `'only-on-failure'` | Useful for CI debugging | +| `video` | `'off'` | `'retain-on-failure'` | Recording slows tests | + +```ts +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + use: { + trace: process.env.CI ? 'on-first-retry' : 'off', + screenshot: process.env.CI ? 'only-on-failure' : 'off', + video: process.env.CI ? 'retain-on-failure' : 'off', + }, +}); +``` + +## Anti-Patterns + +| Don't | Problem | Do Instead | +|-------|---------|------------| +| `timeout: 300_000` globally | Masks flaky tests; slow CI | Fix root cause; keep 30s default | +| Hardcoded URLs: `page.goto('http://localhost:4000/login')` | Breaks in other environments | Use `baseURL` + relative paths | +| All browsers on every PR | 3x CI time | Chromium on PRs; all on main | +| `trace: 'on'` always | Huge artifacts, slow uploads | `trace: 'on-first-retry'` | +| `video: 'on'` always | Massive storage; slow tests | `video: 'retain-on-failure'` | +| Config in test files: `test.use({ viewport: {...} })` everywhere | Scattered, inconsistent | Define once in project config | +| `retries: 3` locally | Hides flakiness | `retries: 0` local, `retries: 2` CI | +| No `forbidOnly` in CI | Committed `test.only` runs single test | `forbidOnly: !!process.env.CI` | +| `globalSetup` for browser auth | No browser context available | Use setup project with dependencies | +| Committing `.env` with credentials | Security risk | Commit `.env.example` only | + +## Troubleshooting + +### baseURL Not Working + +**Cause**: Using absolute URL in `page.goto()` ignores `baseURL`. + +```ts +// Wrong - ignores baseURL +await page.goto('http://localhost:4000/dashboard'); + +// Correct - uses baseURL +await page.goto('/dashboard'); +``` + +### webServer Starts But Tests Get Connection Refused + +**Cause**: `webServer.url` doesn't match actual server address or health check returns non-200. + +```ts +webServer: { + command: 'npm run dev', + url: 'http://localhost:4000/api/health', // use real endpoint + reuseExistingServer: !process.env.CI, + timeout: 120_000, +}, +``` + +### Tests Pass Locally But Timeout in CI + +**Cause**: CI machines are slower. Increase timeouts and reduce workers: + +```ts +export default defineConfig({ + workers: process.env.CI ? '50%' : undefined, + use: { + navigationTimeout: process.env.CI ? 30_000 : 15_000, + actionTimeout: process.env.CI ? 15_000 : 10_000, + }, +}); +``` + +### "Target page, context or browser has been closed" + +**Cause**: Test exceeded `timeout` and Playwright tore down browser during action. + +**Fix**: Don't increase global timeout. Find slow step using trace: + +```bash +npx playwright test --trace on +npx playwright show-report +``` + +## Related + +- [test-tags.md](./test-tags.md) - tagging and filtering tests with `--grep` +- [fixtures-hooks.md](./fixtures-hooks.md) - custom fixtures for per-test state +- [test-suite-structure.md](test-suite-structure.md) - file structure and naming +- [authentication.md](../advanced/authentication.md) - setup projects for shared auth +- [projects-dependencies.md](./projects-dependencies.md) - advanced multi-project patterns diff --git a/external/develop/devops/playwright-best-practices/core/fixtures-hooks.md b/external/develop/devops/playwright-best-practices/core/fixtures-hooks.md new file mode 100644 index 0000000..ff9dc93 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/core/fixtures-hooks.md @@ -0,0 +1,417 @@ +# Fixtures & Hooks + +## Table of Contents + +1. [Built-in Fixtures](#built-in-fixtures) +2. [Custom Fixtures](#custom-fixtures) +3. [Fixture Scopes](#fixture-scopes) +4. [Hooks](#hooks) +5. [Authentication Patterns](#authentication-patterns) +6. [Database Fixtures](#database-fixtures) + +## Built-in Fixtures + +### Core Fixtures + +```typescript +test("example", async ({ + page, // Isolated page instance + context, // Browser context (cookies, localStorage) + browser, // Browser instance + browserName, // 'chromium', 'firefox', or 'webkit' + request, // API request context +}) => { + // Each test gets fresh instances +}); +``` + +### Request Fixture + +```typescript +test("API call", async ({ request }) => { + const response = await request.get("/api/users"); + await expect(response).toBeOK(); + + const users = await response.json(); + expect(users).toHaveLength(5); +}); +``` + +## Custom Fixtures + +### Basic Custom Fixture + +```typescript +// fixtures.ts +import { test as base } from "@playwright/test"; + +// Declare fixture types +type MyFixtures = { + todoPage: TodoPage; + apiClient: ApiClient; +}; + +export const test = base.extend({ + // Fixture with setup and teardown + todoPage: async ({ page }, use) => { + const todoPage = new TodoPage(page); + await todoPage.goto(); + + await use(todoPage); // Test runs here + + // Teardown (optional) + await todoPage.clearTodos(); + }, + + // Simple fixture + apiClient: async ({ request }, use) => { + await use(new ApiClient(request)); + }, +}); + +export { expect } from "@playwright/test"; +``` + +### Fixture with Options + +```typescript +type Options = { + defaultUser: { email: string; password: string }; +}; + +type Fixtures = { + authenticatedPage: Page; +}; + +export const test = base.extend({ + // Define option with default + defaultUser: [ + { email: "test@example.com", password: "pass123" }, + { option: true }, + ], + + // Use option in fixture + authenticatedPage: async ({ page, defaultUser }, use) => { + await page.goto("/login"); + await page.getByLabel("Email").fill(defaultUser.email); + await page.getByLabel("Password").fill(defaultUser.password); + await page.getByRole("button", { name: "Sign in" }).click(); + await use(page); + }, +}); + +// Override in config +export default defineConfig({ + use: { + defaultUser: { email: "admin@example.com", password: "admin123" }, + }, +}); +``` + +### Automatic Fixtures + +```typescript +export const test = base.extend<{}, { setupDb: void }>({ + // Auto-fixture runs for every test without explicit usage + setupDb: [ + async ({}, use) => { + await seedDatabase(); + await use(); + await cleanDatabase(); + }, + { auto: true }, + ], +}); +``` + +## Fixture Scopes + +### Test Scope (Default) + +Created fresh for each test: + +```typescript +test.extend({ + page: async ({ browser }, use) => { + const page = await browser.newPage(); + await use(page); + await page.close(); + }, +}); +``` + +### Worker Scope + +Shared across tests in the same worker (each worker gets its own instance; tests in different workers do not share it): + +```typescript +type WorkerFixtures = { + sharedAccount: Account; +}; + +export const test = base.extend<{}, WorkerFixtures>({ + sharedAccount: [ + async ({ browser }, use) => { + // Expensive setup - runs once per worker + const account = await createTestAccount(); + await use(account); + await deleteTestAccount(account); + }, + { scope: "worker" }, + ], +}); +``` + +### Isolate test data between parallel workers + +When tests in different workers touch the same backend or DB (e.g. same user, same tenant), they can collide and cause flaky failures. Use `testInfo.workerIndex` (or `process.env.TEST_WORKER_INDEX`) in a worker-scoped fixture to create unique data per worker: + +```typescript +import { test as baseTest } from "@playwright/test"; + +type WorkerFixtures = { + dbUserName: string; +}; + +export const test = baseTest.extend<{}, WorkerFixtures>({ + dbUserName: [ + async ({}, use, testInfo) => { + const userName = `user-${testInfo.workerIndex}`; + await createUserInTestDatabase(userName); + await use(userName); + await deleteUserFromTestDatabase(userName); + }, + { scope: "worker" }, + ], +}); +``` + +Then each worker uses a distinct user (e.g. `user-1`, `user-2`), so parallel workers do not overwrite each other’s data. + +## Hooks + +### beforeEach / afterEach + +```typescript +test.beforeEach(async ({ page }) => { + // Runs before each test in file + await page.goto("/"); +}); + +test.afterEach(async ({ page }, testInfo) => { + // Runs after each test + if (testInfo.status !== "passed") { + await page.screenshot({ path: `failed-${testInfo.title}.png` }); + } +}); +``` + +### beforeAll / afterAll + +```typescript +test.beforeAll(async ({ browser }) => { + // Runs once before all tests in file + // Note: Cannot use page fixture here +}); + +test.afterAll(async () => { + // Runs once after all tests in file +}); +``` + +### Describe-Level Hooks + +```typescript +test.describe("User Management", () => { + test.beforeEach(async ({ page }) => { + await page.goto("/users"); + }); + + test("can list users", async ({ page }) => { + // Starts at /users + }); + + test("can add user", async ({ page }) => { + // Starts at /users + }); +}); +``` + +## Authentication Patterns + +### Global Setup with Storage State + +```typescript +// auth.setup.ts +import { test as setup, expect } from "@playwright/test"; + +const authFile = ".auth/user.json"; + +setup("authenticate", async ({ page }) => { + await page.goto("/login"); + await page.getByLabel("Email").fill(process.env.TEST_EMAIL!); + await page.getByLabel("Password").fill(process.env.TEST_PASSWORD!); + await page.getByRole("button", { name: "Sign in" }).click(); + + await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible(); + await page.context().storageState({ path: authFile }); +}); +``` + +```typescript +// playwright.config.ts +export default defineConfig({ + projects: [ + { name: "setup", testMatch: /.*\.setup\.ts/ }, + { + name: "chromium", + use: { + ...devices["Desktop Chrome"], + storageState: ".auth/user.json", + }, + dependencies: ["setup"], + }, + ], +}); +``` + +### Multiple Auth States + +```typescript +// auth.setup.ts +setup("admin auth", async ({ page }) => { + await login(page, "admin@example.com", "adminpass"); + await page.context().storageState({ path: ".auth/admin.json" }); +}); + +setup("user auth", async ({ page }) => { + await login(page, "user@example.com", "userpass"); + await page.context().storageState({ path: ".auth/user.json" }); +}); +``` + +```typescript +// playwright.config.ts +projects: [ + { + name: "admin tests", + testMatch: /.*admin.*\.spec\.ts/, + use: { storageState: ".auth/admin.json" }, + dependencies: ["setup"], + }, + { + name: "user tests", + testMatch: /.*user.*\.spec\.ts/, + use: { storageState: ".auth/user.json" }, + dependencies: ["setup"], + }, +]; +``` + +### Auth Fixture + +```typescript +// fixtures/auth.fixture.ts +export const test = base.extend<{ adminPage: Page; userPage: Page }>({ + adminPage: async ({ browser }, use) => { + const context = await browser.newContext({ + storageState: ".auth/admin.json", + }); + const page = await context.newPage(); + await use(page); + await context.close(); + }, + + userPage: async ({ browser }, use) => { + const context = await browser.newContext({ + storageState: ".auth/user.json", + }); + const page = await context.newPage(); + await use(page); + await context.close(); + }, +}); +``` + +## Database Fixtures + +This section covers **per-test database fixtures** (isolation, transaction rollback). For related topics: + +- **Test data factories** (builders, Faker): See [test-data.md](test-data.md) +- **One-time database setup** (migrations, snapshots): See [global-setup.md](global-setup.md#database-patterns) + +### Transaction Rollback Pattern + +```typescript +import { test as base } from "@playwright/test"; +import { db } from "../db"; + +export const test = base.extend<{ dbTransaction: Transaction }>({ + dbTransaction: async ({}, use) => { + const transaction = await db.beginTransaction(); + + await use(transaction); + + await transaction.rollback(); // Clean slate for next test + }, +}); +``` + +### Seed Data Fixture + +```typescript +type TestData = { + testUser: User; + testProducts: Product[]; +}; + +export const test = base.extend({ + testUser: async ({}, use) => { + const user = await db.users.create({ + email: `test-${Date.now()}@example.com`, + name: "Test User", + }); + + await use(user); + + await db.users.delete(user.id); + }, + + testProducts: async ({ testUser }, use) => { + const products = await db.products.createMany([ + { name: "Product A", ownerId: testUser.id }, + { name: "Product B", ownerId: testUser.id }, + ]); + + await use(products); + + await db.products.deleteMany(products.map((p) => p.id)); + }, +}); +``` + +## Fixture Tips + +| Tip | Explanation | +| ------------------ | ------------------------------------------- | +| Fixtures are lazy | Only created when used | +| Compose fixtures | Use other fixtures as dependencies | +| Keep setup minimal | Do heavy lifting in worker-scoped fixtures | +| Clean up resources | Use teardown in fixtures, not afterEach | +| Avoid shared state | Each fixture instance should be independent | + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ----------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Shared mutable state between tests | Race conditions, order dependencies | Use fixtures for isolation | +| Global variables in tests | Tests depend on execution order | Use fixtures or beforeEach for setup | +| Not cleaning up test data | Tests interfere with each other | Use fixtures with teardown or database transactions | +| Shared `page` or `context` in `beforeAll` | State leak between tests; flaky when tests run in parallel | Use default one-context-per-test, or `beforeEach` + fresh page; if serial is required, prefer `test.describe.configure({ mode: 'serial' })` and document that isolation is sacrificed | +| Backend/DB state shared across workers | Tests in different workers collide on same data | Use worker-scoped fixture with `testInfo.workerIndex` to create unique data per worker | + +## Related References + +- **Page Objects with fixtures**: See [page-object-model.md](page-object-model.md) for POM patterns +- **Test organization**: See [test-suite-structure.md](test-suite-structure.md) for test structure +- **Debugging fixture issues**: See [debugging.md](../debugging/debugging.md) for troubleshooting diff --git a/external/develop/devops/playwright-best-practices/core/global-setup.md b/external/develop/devops/playwright-best-practices/core/global-setup.md new file mode 100644 index 0000000..a033522 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/core/global-setup.md @@ -0,0 +1,434 @@ +# Global Setup & Teardown + +## Table of Contents + +1. [Global Setup](#global-setup) +2. [Global Teardown](#global-teardown) +3. [Database Patterns](#database-patterns) +4. [Environment Provisioning](#environment-provisioning) +5. [Setup Projects vs Global Setup](#setup-projects-vs-global-setup) +6. [Parallel Execution Caveats](#parallel-execution-caveats) + +## Global Setup + +### Basic Global Setup + +```typescript +// global-setup.ts +import { FullConfig } from "@playwright/test"; + +async function globalSetup(config: FullConfig) { + console.log("Running global setup..."); + // Perform one-time setup: start services, run migrations, etc. +} + +export default globalSetup; +``` + +### Configure Global Setup + +```typescript +// playwright.config.ts +import { defineConfig } from "@playwright/test"; + +export default defineConfig({ + globalSetup: require.resolve("./global-setup"), + globalTeardown: require.resolve("./global-teardown"), +}); +``` + +> **Authentication in Global Setup**: For authentication patterns using storage state in global setup, see [fixtures-hooks.md](fixtures-hooks.md#authentication-patterns). Setup projects are generally preferred for authentication as they provide access to Playwright fixtures. + +### Global Setup with Return Value + +```typescript +// global-setup.ts +async function globalSetup(config: FullConfig): Promise<() => Promise> { + const server = await startTestServer(); + + // Return cleanup function (alternative to globalTeardown) + return async () => { + await server.stop(); + }; +} + +export default globalSetup; +``` + +### Access Config in Global Setup + +```typescript +// global-setup.ts +import { FullConfig } from "@playwright/test"; + +async function globalSetup(config: FullConfig) { + const { baseURL } = config.projects[0].use; + console.log(`Setting up for ${baseURL}`); + + // Access custom config + const workers = config.workers; + const timeout = config.timeout; + + // Access environment + const isCI = !!process.env.CI; +} + +export default globalSetup; +``` + +## Global Teardown + +### Basic Global Teardown + +```typescript +// global-teardown.ts +import { FullConfig } from "@playwright/test"; +import fs from "fs"; + +async function globalTeardown(config: FullConfig) { + console.log("Running global teardown..."); + + // Clean up auth files + if (fs.existsSync(".auth")) { + fs.rmSync(".auth", { recursive: true }); + } + + // Clean up test data + await cleanupTestDatabase(); + + // Stop services + await stopTestServices(); +} + +export default globalTeardown; +``` + +### Conditional Teardown + +```typescript +// global-teardown.ts +async function globalTeardown(config: FullConfig) { + // Skip cleanup in CI (containers are discarded anyway) + if (process.env.CI) { + console.log("Skipping teardown in CI"); + return; + } + + // Local cleanup + await cleanupLocalTestData(); +} + +export default globalTeardown; +``` + +## Database Patterns + +This section covers **one-time database setup** (migrations, snapshots, per-worker databases). For related topics: + +- **Per-test database fixtures** (isolation, transaction rollback): See [fixtures-hooks.md](fixtures-hooks.md#database-fixtures) +- **Test data factories** (builders, Faker): See [test-data.md](test-data.md) + +### Database Migration in Setup + +```typescript +// global-setup.ts +import { execSync } from "child_process"; + +async function globalSetup() { + console.log("Running database migrations..."); + + // Run migrations + execSync("npx prisma migrate deploy", { stdio: "inherit" }); + + // Seed test data + execSync("npx prisma db seed", { stdio: "inherit" }); +} + +export default globalSetup; +``` + +### Database Snapshot Pattern + +```typescript +// global-setup.ts +import { execSync } from "child_process"; +import fs from "fs"; + +const SNAPSHOT_PATH = "./test-db-snapshot.sql"; + +async function globalSetup() { + // Check if snapshot exists + if (fs.existsSync(SNAPSHOT_PATH)) { + console.log("Restoring database from snapshot..."); + execSync(`psql $DATABASE_URL < ${SNAPSHOT_PATH}`, { stdio: "inherit" }); + return; + } + + // First run: migrate and create snapshot + console.log("Creating database snapshot..."); + execSync("npx prisma migrate deploy", { stdio: "inherit" }); + execSync("npx prisma db seed", { stdio: "inherit" }); + execSync(`pg_dump $DATABASE_URL > ${SNAPSHOT_PATH}`, { stdio: "inherit" }); +} + +export default globalSetup; +``` + +### Test Database per Worker + +```typescript +// global-setup.ts +async function globalSetup(config: FullConfig) { + const workerCount = config.workers || 1; + + // Create a database for each worker + for (let i = 0; i < workerCount; i++) { + const dbName = `test_db_worker_${i}`; + await createDatabase(dbName); + await runMigrations(dbName); + await seedDatabase(dbName); + } +} + +// global-teardown.ts +async function globalTeardown(config: FullConfig) { + const workerCount = config.workers || 1; + + for (let i = 0; i < workerCount; i++) { + await dropDatabase(`test_db_worker_${i}`); + } +} +``` + +## Environment Provisioning + +### Start Services in Setup + +```typescript +// global-setup.ts +import { execSync, spawn } from "child_process"; + +let serverProcess: any; + +async function globalSetup() { + // Start backend server + serverProcess = spawn("npm", ["run", "start:test"], { + stdio: "pipe", + detached: true, + }); + + // Wait for server to be ready + await waitForServer("http://localhost:3000/health", 30000); + + // Store PID for teardown + process.env.SERVER_PID = serverProcess.pid.toString(); +} + +async function waitForServer(url: string, timeout: number) { + const start = Date.now(); + + while (Date.now() - start < timeout) { + try { + const response = await fetch(url); + if (response.ok) return; + } catch { + // Server not ready yet + } + await new Promise((r) => setTimeout(r, 1000)); + } + + throw new Error(`Server did not start within ${timeout}ms`); +} + +export default globalSetup; +``` + +### Docker Compose Setup + +```typescript +// global-setup.ts +import { execSync } from "child_process"; + +async function globalSetup() { + console.log("Starting Docker services..."); + + execSync("docker-compose -f docker-compose.test.yml up -d", { + stdio: "inherit", + }); + + // Wait for services to be healthy + execSync("docker-compose -f docker-compose.test.yml exec -T db pg_isready", { + stdio: "inherit", + }); +} + +export default globalSetup; +``` + +```typescript +// global-teardown.ts +import { execSync } from "child_process"; + +async function globalTeardown() { + console.log("Stopping Docker services..."); + + execSync("docker-compose -f docker-compose.test.yml down -v", { + stdio: "inherit", + }); +} + +export default globalTeardown; +``` + +### Environment Variables Setup + +```typescript +// global-setup.ts +import dotenv from "dotenv"; +import path from "path"; + +async function globalSetup() { + // Load test-specific environment + const envFile = process.env.CI ? ".env.ci" : ".env.test"; + dotenv.config({ path: path.resolve(process.cwd(), envFile) }); + + // Validate required variables + const required = ["DATABASE_URL", "API_KEY", "TEST_EMAIL"]; + for (const key of required) { + if (!process.env[key]) { + throw new Error(`Missing required environment variable: ${key}`); + } + } +} + +export default globalSetup; +``` + +## Setup Projects vs Global Setup + +### When to Use Each + +| Use Global Setup | Use Setup Projects | +| ------------------------------------- | ---------------------------------------- | +| One-time setup (migrations, services) | Per-project setup (auth states) | +| No access to Playwright fixtures | Need page, request fixtures | +| Runs once before all projects | Can run per-project or have dependencies | +| Shared across all workers | Can be parallelized | + +### Setup Project Pattern + +```typescript +// playwright.config.ts +export default defineConfig({ + projects: [ + // Setup project + { + name: "setup", + testMatch: /.*\.setup\.ts/, + }, + // Test projects depend on setup + { + name: "chromium", + use: { ...devices["Desktop Chrome"] }, + dependencies: ["setup"], + }, + { + name: "firefox", + use: { ...devices["Desktop Firefox"] }, + dependencies: ["setup"], + }, + ], +}); +``` + +> **For complete authentication setup patterns**, see [fixtures-hooks.md](fixtures-hooks.md#authentication-patterns). + +### Combining Both + +```typescript +// playwright.config.ts +export default defineConfig({ + // Global: Start services, run migrations + globalSetup: require.resolve("./global-setup"), + globalTeardown: require.resolve("./global-teardown"), + + projects: [ + // Setup project: Create auth states + { name: "setup", testMatch: /.*\.setup\.ts/ }, + { + name: "chromium", + use: { + ...devices["Desktop Chrome"], + storageState: ".auth/user.json", + }, + dependencies: ["setup"], + }, + ], +}); +``` + +## Parallel Execution Caveats + +### Understanding Global Setup Execution + +``` +┌─────────────────────────────────────────────────────────────┐ +│ globalSetup runs ONCE │ +│ ↓ │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ +│ │ Worker 1│ │ Worker 2│ │ Worker 3│ │ Worker 4│ │ +│ │ tests │ │ tests │ │ tests │ │ tests │ │ +│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ +│ ↓ │ +│ globalTeardown runs ONCE │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Key implications:** + +- Global setup has **no access** to Playwright fixtures (`page`, `request`, `context`) +- State created in global setup is **shared** across all workers +- If tests **modify** shared state, they may conflict with parallel workers +- Global setup **cannot** react to individual test needs + +### When to Prefer Worker-Scoped Fixtures + +Use **worker-scoped fixtures** instead of globalSetup when: + +| Scenario | Why Fixtures Are Better | +| ------------------------------------ | ---------------------------------------------------- | +| Each worker needs isolated resources | Fixtures can create per-worker databases, servers | +| Setup needs Playwright APIs | Fixtures have access to `page`, `request`, `browser` | +| Setup depends on test configuration | Fixtures receive test context and options | +| Resources need cleanup per worker | Worker fixtures auto-cleanup when worker exits | + +### Common Parallel Pitfall + +```typescript +// ❌ BAD: Global setup creates ONE user, all workers fight over it +async function globalSetup() { + await createUser({ email: "test@example.com" }); // Shared! +} + +// ✅ GOOD: Each worker gets its own user via worker-scoped fixture +// Uses workerInfo.workerIndex to create unique data per worker +``` + +> **For worker-scoped fixture patterns** (per-worker databases, unique test data, `workerIndex` isolation), see [fixtures-hooks.md](fixtures-hooks.md#isolate-test-data-between-parallel-workers). + +## Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Solution | +| ------------------------------ | -------------------------------- | ------------------------------------------ | +| Heavy setup in globalSetup | Slow test startup | Use setup projects for parallelizable work | +| Not cleaning up in teardown | Leaks resources, flaky CI | Always clean up or use containers | +| Hardcoded URLs in setup | Breaks in different environments | Use config.projects[0].use.baseURL | +| No timeout on service wait | Hangs forever if service fails | Add timeout with clear error | +| Shared mutable state | Race conditions in parallel | Use worker-scoped fixtures for isolation | +| Global setup for per-test data | Tests conflict | Use test-scoped fixtures | + +## Related References + +- **Fixtures & Auth**: See [fixtures-hooks.md](fixtures-hooks.md) for worker-scoped fixtures and auth patterns +- **CI/CD**: See [ci-cd.md](../infrastructure-ci-cd/ci-cd.md) for CI setup patterns +- **Projects**: See [projects-dependencies.md](projects-dependencies.md) for project configuration diff --git a/external/develop/devops/playwright-best-practices/core/locators.md b/external/develop/devops/playwright-best-practices/core/locators.md new file mode 100644 index 0000000..f806635 --- /dev/null +++ b/external/develop/devops/playwright-best-practices/core/locators.md @@ -0,0 +1,242 @@ +# Locator Strategies + +## Table of Contents + +1. [Priority Order](#priority-order) +2. [User-Facing Locators](#user-facing-locators) +3. [Filtering & Chaining](#filtering--chaining) +4. [Dynamic Content](#dynamic-content) +5. [Shadow DOM](#shadow-dom) +6. [Iframes](#iframes) + +## Priority Order + +Use locators in this order of preference: + +1. **Role-based** (most resilient): `getByRole` +2. **Label-based**: `getByLabel`, `getByPlaceholder` +3. **Text-based**: `getByText`, `getByTitle` +4. **Test IDs** (when semantic locators aren't possible): `getByTestId` +5. **CSS/XPath** (last resort): `locator('css=...')`, `locator('xpath=...')` + +## User-Facing Locators + +### getByRole + +Most robust approach - matches how users and assistive technology perceive the page. + +```typescript +// Buttons +page.getByRole("button", { name: "Submit", exact: true }); // exact accessible name +page.getByRole("button", { name: /submit/i }); // flexible case-insensitive match + +// Links +page.getByRole("link", { name: "Home" }); + +// Form elements +page.getByRole("textbox", { name: "Email" }); +page.getByRole("checkbox", { name: "Remember me" }); +page.getByRole("combobox", { name: "Country" }); +page.getByRole("radio", { name: "Option A" }); + +// Headings +page.getByRole("heading", { name: "Welcome", level: 1 }); + +// Lists & items +page.getByRole("list").getByRole("listitem"); + +// Navigation & regions +page.getByRole("navigation"); +page.getByRole("main"); +page.getByRole("dialog"); +page.getByRole("alert"); +``` + +### getByLabel + +For form elements with associated labels. + +```typescript +// Input with