@testchimp/playwright

A Playwright / Mobilewright reporter that sends test execution results to the TestChimp platform. It powers QA intelligence insights, surfaces AI-native steps (e.g. ai.act, ai.verify) from the test runner into TestChimp for CI, and augments RUM events from @testchimp/rum-js so test runs align with real user events for TrueCoverage.

Purpose

1. Report execution results to TestChimp for QA intelligence

The reporter collects test runs (pass/fail, steps, errors, timing) and sends them to the TestChimp backend. TestChimp uses this data to:

Track which tests ran, when, and their outcome
Drive dashboards, trends, and QA intelligence
Correlate failures with steps and screenshots for faster debugging
Support traceability between tests, scenarios, and coverage

You run tests with the normal Playwright CLI or via your CI (GitHub Actions etc.); the reporter runs in process and posts results to TestChimp without changing how you execute tests.

2. Pipe AI-native steps through TestChimp so they work wherever you run tests

The reporter plugin pipes AI-native step calls (ai.act, ai.verify, etc.) via TestChimp backends, so that those steps work seamlessly—wherever you run your tests (local, CI, or any environment).

3. Augment RUM events for TrueCoverage (test ↔ real user alignment)

@testchimp/rum-js (web), TestChimpRum (iOS — Swift package testchimp-rum-ios), and TestChimpRum (Android — testchimp-rum-android, typically via JitPack: com.github.testchimphq:testchimp-rum-android:<tag> per JitPack) emit real user events to TestChimp. When the same app is exercised in CI (Playwright or Mobilewright), you want those events tagged with which test produced them so TestChimp can:

Align test runs with real user sessions (TrueCoverage)
See which tests generated which events
Compare test coverage to production usage to drive better QA strategy.

Read more about TestChimps' TrueCoverage feature here.

Installation

npm install @testchimp/playwright

Peer dependency: @playwright/test (e.g. >=1.40.0) for web projects, and mobilewright for mobile projects.

Platform is determined per test from Mobilewright projects[].use.platform (ios / android; omitted = web), with fallback to testInfo.annotations (device.platform, Mobilewright 0.0.36+). Requires mobilewright >= 0.0.37 for multi-project installApps. Pass installTestChimp(base, { uiFixture: 'screen' }) when wrapping @mobilewright/test; default page for @playwright/test (web path never touches device fixtures).

Quick start

1. Playwright config

Configure the reporter in your playwright.config.js like below:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

// Optional: import runtime so CI test info is injected for @testchimp/rum-js (TrueCoverage)


export default defineConfig({
  reporter: [
    ['list'],
    ['@testchimp/playwright/reporter', {
      verbose: true,
      reportOnlyFinalAttempt: true,
      captureScreenshots: true,
    }],
  ],
});

For TrueCoverage, wrap and re-export test from your fixtures entry (recommended):

import { test as base } from '@playwright/test';
import { installTestChimp } from '@testchimp/playwright/runtime';
export const test = installTestChimp(base); // default uiFixture: 'page'

// Mobilewright fixtures barrel:
// import { test as base } from '@mobilewright/test';
// export const test = installTestChimp(base, { uiFixture: 'screen' });

A side-effect-only import '@testchimp/playwright/runtime' does not apply extended fixtures or mobile hooks; the returned TestType from installTestChimp must be what your specs import.

Web: the runtime injects __TC_CI_TEST_INFO on the page fixture for @testchimp/rum-js, then flushes in page fixture teardown (after all afterEach hooks, while the page is still open): re-sync CI, briefly poll for buffered events, then await globalThis.__TC_RUM_FLUSH() (rum-js ≥ 0.1.7). Requires @testchimp/playwright ≥ 0.2.6. Optional: TESTCHIMP_RUM_WEB_FLUSH_TIMEOUT_MS (default 5000), TESTCHIMP_RUM_WEB_BUFFER_POLL_MS (default 500, max 2000), TESTCHIMP_RUM_WEB_FLUSH_DEBUG=1.

Mobile (iOS/Android): set use.platform to ios or android on Mobilewright UI projects and use installTestChimp(base, { uiFixture: 'screen' }). The runtime extends the Mobilewright device fixture so SET runs right after Mobilewright’s launchApp (via device.openUrl), before screen and the test body; afterEach still sends a trailing SET + v1/flush. By default no /v1/clear between tests—each test SETs new CI so RUM avoids a clear→set gap with missing ci_test_info. iOS: testchimp-rum-ios ≥ 0.1.5 starts a new RUM session on that first per-process SET (when CI was not yet active), so rum_sessions get environment / release without extra runner URLs. Integrate TestChimpRum for that platform (see testchimp-rum-ios / testchimp-rum-android READMEs): URL scheme / intent filter for testchimp-rum://truecoverage/....

Optional URL overrides: TESTCHIMP_RUM_AUTOMATION_SET_PREFIX, TESTCHIMP_RUM_AUTOMATION_CLEAR_URL, TESTCHIMP_RUM_AUTOMATION_FLUSH_URL (defaults match the native SDKs). Legacy: set TESTCHIMP_RUM_AUTOMATION_CLEAR_BETWEEN_TESTS=1 to send /v1/clear before each test’s SET again.

2. Environment variables

Set these so the reporter can talk to TestChimp (env vars override programmatic options):

Variable	Required	Description
`TESTCHIMP_API_KEY`	Yes	API key for TestChimp.
`TESTCHIMP_PROJECT_ID`	No	Legacy/optional; the backend resolves the project from the API key when omitted.
`TESTCHIMP_TESTS_FOLDER`	No	Base folder for relative paths (default: `tests`).
`TESTCHIMP_RELEASE`	No	Release/version identifier.
`TESTCHIMP_ENV`	No	Environment (e.g. `staging`, `prod`).
`TESTCHIMP_EXECUTION_SOURCE`	No	Run source stamped on ingest: `CI`, `LOCAL_AGENT`, `CLOUD_AGENT`, or `MANUAL`. Skill/agent runs set `LOCAL_AGENT` / `CLOUD_AGENT`. True CI pipelines set `CI`. If unset: remote cloud-agent host (`COPILOT_USE_PLATFORM` / `COPILOT_WORKSPACE` / `CURSOR_AGENT_WORKER_ID`) → `CLOUD_AGENT`; local agent host (`CURSOR_AGENT`, Claude Code, …) → `LOCAL_AGENT`; else `CI=true`/`1` → `CI`; else `LOCAL_AGENT`. `CURSOR_AGENT` alone is the local Cursor IDE — not cloud. Not the same as `TESTCHIMP_EXECUTION_MODE` (ingest path).
`TESTCHIMP_ENABLE_API_CAPTURE`	No	Set to `1`/`true`/`yes` to opt in to API operation capture + payload ingest (default off).
`TESTCHIMP_API_COVERAGE_URL_REGEX`	No	Optional client-side URL allow-list regex (full URL).
`TESTCHIMP_API_COVERAGE_BODY_TIMEOUT_MS`	No	Per-response body read timeout in ms (default 1500).
`TESTCHIMP_API_COVERAGE_DRAIN_TIMEOUT_MS`	No	Page teardown drain timeout in ms (default 5000).
`TESTCHIMP_RUM_AUTOMATION_SET_PREFIX`	No	Override set-URL prefix (default `testchimp-rum://truecoverage/v1/set?p=` + base64url JSON).
`TESTCHIMP_RUM_AUTOMATION_CLEAR_URL`	No	Override clear URL (default `testchimp-rum://truecoverage/v1/clear`).
`TESTCHIMP_RUM_AUTOMATION_CLEAR_BETWEEN_TESTS`	No	When `1`/`true`/`yes`, each test’s `device` fixture sends `/v1/clear` before `SET` (legacy). Default off so CI stays set until overwritten, avoiding null `ci_test_info` on RUM during the clear→set window.
`TESTCHIMP_RUM_AUTOMATION_SUITE_TEARDOWN_CLEAR`	No	When `1`/`true`/`yes`, registers `afterAll` to send `/v1/clear` then `/v1/flush` after the spec file. Default off (avoids clearing CI between spec files before the next file’s `SET`).
`TESTCHIMP_RUM_TRANSPORT_RESYNC`	No	Mobile `screen` fixture: set to `0` to disable automatic TrueCoverage `v1/set` after likely WebSocket/mobilecli transport failures (default: enabled).
`TESTCHIMP_RUM_WEB_FLUSH_TIMEOUT_MS`	No	Web `page` fixture: per-call timeout for RUM flush `page.evaluate` (default 5000, clamp 100–30000).

If TESTCHIMP_API_KEY is missing, the reporter logs a warning and disables reporting (no request is sent).

API operation capture (Operations coverage)

Off by default. Set TESTCHIMP_ENABLE_API_CAPTURE=1 to enable both HTTP interaction capture (via the web page fixture from installTestChimp) and reporter ingest of those interactions. One flag controls both; leave it unset so the plugin never attaches network listeners or reads response bodies.

export TESTCHIMP_API_KEY=your-api-key
export TESTCHIMP_ENABLE_API_CAPTURE=1
npx playwright test

Capture is best-effort and bounded (body / drain timeouts above) so it must not hang test teardown. Native Playwright mocks registered with page.route(...) / context.route(...) after the TestChimp page fixture attaches, and completed with route.fulfill(...), are automatically classified as mocked. Routes that continue/fallback/abort to the network remain real.

Mock classification is also best-effort:

Routes registered before the page fixture attaches are not instrumented. Use fulfillMocked(route, options) (or stamp x-testchimp-mocked: 1) for those handlers.
routeFromHAR(...) after attach that fulfills through Playwright routing is classified as mocked.
Service-worker-based tools such as MSW, and non-Playwright mocking layers, may not be identifiable and therefore default to real.

Classification never changes fulfill options or blocks a route. Project Operations config can further restrict which URLs are stored and whether payloads are retained.

3. Run tests

export TESTCHIMP_API_KEY=your-api-key
npx playwright test

Results are reported to TestChimp after each test (or after the final attempt when using retries and reportOnlyFinalAttempt: true).

Reporter options

You can pass options in playwright.config.ts:

['@testchimp/playwright/reporter', {
  apiKey: '...',           // override env (not recommended in CI)
  ingressUrl: '...',       // override TESTCHIMP_INGRESS_URL (CI ingest host)
  backendUrl: '...',       // override TESTCHIMP_BACKEND_URL (control plane / ai-wright)
  projectId: '...',       // optional override env
  testsFolder: 'tests',   // base dir for relative path calculation
  release: '1.0.0',
  environment: 'staging',
  reportOnlyFinalAttempt: true,  // only send report for last retry (default: true)
  captureScreenshots: true,      // attach screenshots to failing steps (default: true)
  verbose: false,               // extra logging (default: false)
}]

Environment variables take precedence over these options.

CI ingest host: default https://ingress.testchimp.io. Set TESTCHIMP_INGRESS_URL (or ingressUrl) to override. If only TESTCHIMP_BACKEND_URL is set to a SaaS featureservice*.testchimp.io host, the reporter rewrites it to the matching ingress* host for ingest. Keep TESTCHIMP_BACKEND_URL as featureservice for CLI / MCP / ai-wright.

What gets reported

For each test (or its final attempt when using retries), the reporter sends a Smart Test Execution Report that includes:

Identity: folderPath, fileName, suitePath, testName (derived from test file and describe blocks).
Run context: batchInvocationId, branchName (from env when available), release, environment.
Job detail:
- Steps: Every Playwright step with category test.step, expect, or pw:api (including AI-native steps).
- Status: Completed, Failed, or Unknown (mapped from Playwright status).
- Error: Top-level test error message if failed.
- Screenshots: For failing steps, when captureScreenshots is true and Playwright has attached screenshots (e.g. screenshot: 'only-on-failure').

Retries are tracked; with reportOnlyFinalAttempt: true only the last attempt is reported.

Exports

Subpath: @testchimp/playwright/reporter — explicit reporter entry for Playwright reporter config.
Named: TestChimpReporter, TestChimpApiClient, and types/utilities from ./types and ./utils.
Subpath: @testchimp/playwright/runtime — use installTestChimp(test) on your runner’s test object (see Quick start).
- Web: extends page for __TC_CI_TEST_INFO; afterEach flushes via globalThis.__TC_RUM_FLUSH (rum-js ≥ 0.1.3).
- Mobile: when projects[].use.platform is ios/android, registers hooks that call device.openUrl for the iOS Swift SDK and Android Kotlin SDK (same URL contract). Default: SET-only between tests; no automatic afterAll clear (CI clears on SDK TTL or opt-in TESTCHIMP_RUM_AUTOMATION_SUITE_TEARDOWN_CLEAR=1).
Named: buildCiTestInfoJson, attachMobileRumAutomationHooks, extendMobileTestWithTrueCoverageDevice, clearBetweenTestsEnabled, suiteTeardownClearEnabled, resyncTrueCoverageSetForCurrentTest, isLikelyMobileTransportFailure, etc., for advanced wiring.

Troubleshooting

“Reporting disabled”
Set TESTCHIMP_API_KEY (or pass it in reporter options). TESTCHIMP_PROJECT_ID should be present to enable TrueCoverage.
No steps or only some steps
Only steps with category test.step, expect, or pw:api are reported. Internal/hook steps are excluded.
No screenshots on failure
Enable screenshot capture in Playwright (e.g. use: { screenshot: 'only-on-failure' }). The reporter only attaches existing attachments to failing steps.
RUM events not linked to tests / missing ci_test_info (mobile)
Use export const test = installTestChimp(base, { uiFixture: 'screen' }) from a Mobilewright fixtures barrel and import test from there. Set use.platform on Mobilewright projects and handle testchimp-rum://truecoverage/... in the app (TestChimpRum.handleAutomationURL on iOS, TestChimpRum.handleAutomationUri on Android).
Android SDK: use testchimp-rum-android ≥ 0.1.7 (automation /v1/set on caller thread; /v1/flush drains buffered events when the runner opens that URL).
Runner: installTestChimp(..., { uiFixture: 'screen' }) extends the device fixture so one v1/set runs right after Mobilewright’s launchApp (before screen), then settle (TESTCHIMP_RUM_AUTOMATION_POST_SET_SETTLE_MS, default 100 ms). afterEach (mobile projects only) sends a trailing set + v1/flush. Does not send /v1/clear between tests by default (avoids null ci-test-info). Opt in to legacy clear-first with TESTCHIMP_RUM_AUTOMATION_CLEAR_BETWEEN_TESTS=1, or TESTCHIMP_RUM_AUTOMATION_SUITE_TEARDOWN_CLEAR=1 for afterAll clear+flush after each spec file. Re-sends set after likely transport failures on screen API calls. Disable resync with TESTCHIMP_RUM_TRANSPORT_RESYNC=0. Optional URL overrides: TESTCHIMP_RUM_AUTOMATION_FLUSH_URL. If device.openUrl wedges (mobilecli), set TESTCHIMP_RUM_AUTOMATION_OPEN_URL_TIMEOUT_MS (default 25000, clamp 100–120000). Use @testchimp/playwright ≥ 0.2.3 (restores 0.1.43 device SET behaviour; 0.2.0–0.2.2 could skip SET when platformFromTestInfo was web at fixture time).
RUM events not linked to tests (web)
Same installTestChimp barrel; web uses __TC_CI_TEST_INFO on page for @testchimp/rum-js. Use @testchimp/playwright ≥ 0.2.4 for web afterEach flush (no app/test flush code). rum-js keeps normal prod batching; the plugin drains the buffer after each test.
Verbose logging
Set verbose: true in reporter options or use it during setup to see which steps are captured and when reports are sent.

How this helps in real testing scenarios

@testchimp/playwright is the bridge between Playwright / Mobilewright CI and TestChimp: execution reports for QA intelligence, AI-native step routing (ai.act, ai.verify via ai-wright), and TrueCoverage tagging so RUM events from tests align with production usage.

Core docs: TrueCoverage intro · How TrueCoverage works · SmartTests intro · Run SmartTests in CI · Mobile testing

Payments & billing (automate + tag RUM dimensions)

Scenario	Testing guide
Stripe Checkout, Elements, 3DS	Stripe payments in Playwright
Stripe webhooks & idempotency	Stripe webhooks in CI
Apple Pay / Google Pay / PayPal	Wallet payment flows
Subscriptions, trials, dunning	Subscription billing
E-commerce checkout	Checkout flows
Cart, coupons, promos	Cart & promo codes
Tax / VAT / regional pricing	Tax & regional pricing
Refunds & partial credits	Returns & refunds
Trial → paid conversion	Trial to paid
Seat limits & team growth	Seat licensing
Plan entitlements	Feature entitlements

Auth & identity (fixtures, seeds, session probes)

Scenario	Testing guide
Firebase Auth emulator & custom tokens	Firebase authentication
Auth0 / Okta enterprise SSO	Auth0 & Okta SSO
Google / GitHub OAuth	OAuth social login
Magic links & passwordless	Magic link testing
MFA / TOTP / SMS OTP	MFA & 2FA flows
CAPTCHA on signup/login	CAPTCHA-enabled flows
RBAC × permission matrices	RBAC permissions
Admin dashboards	Admin RBAC flows
Session timeout & refresh	Session expiry testing

AI & conversational UX (use `ai.act` / `ai.verify` in SmartTests)

Scenario	Testing guide
Chatbots & multi-turn UI	Conversational UI testing
AI agent tool calling	AI agent workflows
RAG / knowledge-base search	RAG testing
LLM output / JSON schema	LLM output validation
Streaming responses	Streaming AI responses
Evals vs deterministic E2E	AI testing guides hub · Conversational UI
Canvas, charts, maps	Canvas & visual widgets · Google Maps
Hybrid SmartTests	AI-powered web apps

Integrations & async side effects

Scenario	Testing guide
Transactional email (Mailtrap patterns)	Transactional email
SMS / OTP verification	SMS & OTP testing
Async webhooks	Webhooks & async events
PDF invoices & downloads	PDF generation & downloads
File upload & CSV import/export	File uploads · CSV import/export
Third-party iframes & embeds	Third-party embeds
Push notification preferences	Push notifications

UI patterns, SaaS & industry verticals

Scenario	Testing guide
Onboarding funnels & screen states	SaaS onboarding · Screen-state annotations
Form validation & a11y errors	Form validation
Localization & RTL	Localization / i18n
GDPR export / delete / consent	GDPR privacy flows
Search, filters, data grids	Search & filters · Data grids
Calendar & scheduling	Calendar scheduling
Fintech transfers & ledgers	Fintech web apps
HR, healthcare, insurance, legal	HR applications · Healthcare portals · Insurance quotes · E-signatures
Audit & compliance logs	Audit & compliance
Flaky E2E at startups	Fix flaky E2E tests

TestChimp workflow (plans → CI → evolve)

Test planning in Git — markdown scenarios as source of truth
Requirement traceability — // @Scenario: links in Playwright specs
Test runs — CI + manual runs in one view
Arrange / Act / Assert pattern — seeds, probes, hybrid AI steps
/testchimp test · /testchimp evolve · ExploreChimp
Record-replay vs TestChimp

Browse all scenario guides: Testing guides hub · Business flows · Auth · AI UX · Integrations · UI patterns · Verticals

License

MIT.

Name		Name	Last commit message	Last commit date
esm		esm
src		src
test		test
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json
testchimp-playwright-0.2.12.tgz		testchimp-playwright-0.2.12.tgz
testchimp-playwright-0.2.14.tgz		testchimp-playwright-0.2.14.tgz
testchimp-playwright-0.2.15.tgz		testchimp-playwright-0.2.15.tgz
tsconfig.json		tsconfig.json

Navigation Menu

Repository navigation

About

Repository files navigation

@testchimp/playwright

Purpose

1. Report execution results to TestChimp for QA intelligence

2. Pipe AI-native steps through TestChimp so they work wherever you run tests

3. Augment RUM events for TrueCoverage (test ↔ real user alignment)

Installation

Quick start

1. Playwright config

2. Environment variables

API operation capture (Operations coverage)

3. Run tests

Reporter options

What gets reported

Exports

Troubleshooting

How this helps in real testing scenarios

Payments & billing (automate + tag RUM dimensions)

Auth & identity (fixtures, seeds, session probes)

AI & conversational UX (use `ai.act` / `ai.verify` in SmartTests)

Integrations & async side effects

UI patterns, SaaS & industry verticals

TestChimp workflow (plans → CI → evolve)

License

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

@testchimp/playwright

Purpose

1. Report execution results to TestChimp for QA intelligence

2. Pipe AI-native steps through TestChimp so they work wherever you run tests

3. Augment RUM events for TrueCoverage (test ↔ real user alignment)

Installation

Quick start

1. Playwright config

2. Environment variables

API operation capture (Operations coverage)

3. Run tests

Reporter options

What gets reported

Exports

Troubleshooting

How this helps in real testing scenarios

Payments & billing (automate + tag RUM dimensions)

Auth & identity (fixtures, seeds, session probes)

AI & conversational UX (use ai.act / ai.verify in SmartTests)

Integrations & async side effects

UI patterns, SaaS & industry verticals

TestChimp workflow (plans → CI → evolve)

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

AI & conversational UX (use `ai.act` / `ai.verify` in SmartTests)