feat(conformance): generate the adapter capability matrix from one declared source

.github/pull_request_template.md

@@ -29,6 +29,10 @@
 - [ ] `script` (page-injected runtime)
 - [ ] `trace` (Trace mode)
 
+## Capability matrix
+
+- [ ] Capability matrix updated (`pnpm docs:capabilities`) if this adds or changes user-facing behaviour
+
 ## Notes for reviewers
 
 [//]: # (Anything non-obvious: design trade-offs, alternatives you ruled out, follow-ups. If this PR touches more than one adapter, say why the shared logic doesn't live in `core`.)

CLAUDE.md

@@ -29,6 +29,7 @@ Run from repo root unless noted.
 | `pnpm demo:wdio` / `pnpm demo:nightwatch` / `pnpm demo:selenium` | Run the per-framework example projects. Useful for manual verification of UI or runtime changes. **No package pins a browser driver**, deliberately: a pinned `chromedriver` rots against whatever Chrome the developer has, and because pnpm puts the package's `node_modules/.bin` on `PATH`, Selenium Manager finds it there, prefers it over resolving, and on a major mismatch only *warns* before returning it anyway. The session then fails to start. With nothing on `PATH` both adapters resolve a matching driver (Nightwatch's Chrome service builder sets `requiresDriverBinary: false` and passes an unset `server_path` through to the same resolver), and nightwatch declares `chromedriver` an optional peer. |
 | `pnpm demo:wdio:mobile` / `:selenium:mobile` / `:nightwatch:mobile` / `:python:mobile` | The same, against Appium. All four build the same capability bag and drive the Clock app that ships with every Android system image (keying a duration on the timer keypad, never starting a timer), so a native example needs no `.apk`. `DEVTOOLS_MOBILE_PLATFORM=ios` runs the iOS spec instead — all four adapters — which drives Settings because the simulator ships no Clock, from a separate spec per platform rather than a branch. The simulator is chosen by udid and defaults to whichever is already booted, and an unmatched `IOS_DEVICE_NAME` is refused: naming one that does not exist makes the XCUITest driver create and boot it, every run, rather than fail. That refusal and the booted-simulator preflight are **local** policy — `xcrun simctl` enumerates local simulators and nothing else — so `IOS_UDID` and a non-local `APPIUM_HOST` both bypass them, or a real device and a cloud grid would be refused a run they were correctly configured for. `DEVTOOLS_MOBILE=web` drives the device's own browser on both platforms — Chrome on Android, Safari on iOS, which the XCUITest driver serves without a chromedriver. `examples/MOBILE.md` holds the prerequisites and the `DEVTOOLS_MOBILE` / `APPIUM_APP` switches; `DEVTOOLS_MODE=trace` flips any demo to trace mode. |
 | `pnpm dev` | Run all packages in parallel dev mode. |
+| `pnpm docs:capabilities` | Render the per-adapter capability tables from `packages/conformance/src/capabilities.ts` into `README.md`; `--website <webdriverio checkout>` also renders the docs-site tables, and `--check` fails instead of writing. |
 | `python3 packages/selenium-devtools-py/scripts/changes.py next-version` | The version a Python-adapter release would publish, from the fragments pending in `changes/`. `check --base <ref>` is the CI gate; `apply` is what the release runs. |
 
 `selenium-devtools` exposes per-runner variants of its example via `pnpm --filter @wdio/selenium-devtools example:mocha` / `:mocha:allure` / `:jest` / `:cucumber`.
@@ -48,6 +49,7 @@ Defined in root `tsconfig.json`:
 | `@wdio/devtools-script` / `*` | `packages/script/src/...` |
 | `@wdio/devtools-service` / `*` | `packages/service/src/...` |
 | `@wdio/selenium-devtools` / `*` | `packages/selenium-devtools/src/...` |
+| `@wdio/nightwatch-devtools/*` | `packages/nightwatch-devtools/src/*` (subpaths only, for type imports such as the option types) |
 | `@wdio/devtools-shared` / `*` | `packages/shared/src/...` |
 | `@wdio/devtools-core` / `*` | `packages/core/src/...` |
 | `@wdio/devtools-trace` / `*` | `packages/trace/src/...` |
@@ -259,6 +261,7 @@ Neither is hand-versioned: both assemble the version and the changelog at releas
 ### Documentation
 
 - User-facing docs live in two places that must stay in sync: this repo's `README.md` (+ per-package READMEs) and the **WebdriverIO devtools webpage** (`website/docs/devtools/**` in the `webdriverio/webdriverio` repo — e.g. `wdio/TraceMode.md`). When a change adds, removes, or alters user-facing behavior (a new option, CLI, flag, output, or workflow), update the README here **and** mirror it to the matching webpage doc in the same change. A docs PR that updates only one side isn't complete.
+- The capability tables (the README's support, dashboard and mobile tables, the docs site's `CrossFrameworkSupport.md` and `MobileTesting.md`) are generated, never edited by hand. `packages/conformance/src/capabilities.ts` declares every capability with a cell per adapter, typed as a complete map over the conformance suite's adapter ids, so a new adapter does not compile until each capability declares it. A partial or unsupported cell carries a required reason, which renders as the footnote. A cell is `proven` only when it names conformance rules the adapter has no known gap against; it renders as "✅ tested", every other mark is a declaration. Two coverage checks keep a new feature from bypassing the matrix: every user-facing option (`capability-options.ts`, keyed on the real option types of the shared base and each JS adapter, plus the Python `enable()` keywords, pytest options and user-facing `DEVTOOLS_*` variables read from source by a test) names a capability row or says why it is not one, and every conformance rule backs a proven cell or is listed in `INTERNAL_RULES` with a reason. A test fails when the README tables are out of date; the docs-site copy is refreshed with `pnpm docs:capabilities --website <checkout>` in the same change.
 
 ---
 

README.md

@@ -40,6 +40,45 @@ Want a portable artifact instead of a live UI (CI / agent diffing)? Switch to **
 - **Jasmine**: Complete integration with grep-based filtering
 - **Cucumber**: Scenario-level and example-specific execution with feature:line targeting
 
+**Dashboard and options by adapter:**
+
+<!-- capability-matrix:readme-dashboard:start -->
+<!-- Generated by `pnpm docs:capabilities` from packages/conformance/src/capabilities.ts. Do not edit by hand. -->
+
+| Capability | WebdriverIO | Selenium | Nightwatch | Python |
+|---|---|---|---|---|
+| One row per command in the Actions list | ✅ tested | ✅ tested | ✅ tested | ✅ tested |
+| Screenshot on every command row | ✅ | ✅ | ✅ | ✅ |
+| Browser console logs | ✅ | ✅ | ✅ | ✅ |
+| Source tab opens the test line that ran the command | ✅ | ✅ | ✅ | ✅ |
+| Metadata tab (session, capabilities, device) | ✅ | ✅ | ✅ | ✅ |
+| Performance timings on navigation rows | ✅ | ✅ | ✅ | ✅ |
+| Failed command in the Errors tab, with the browser's message | ✅ tested | ✅ tested | ✅ tested | ✅ tested |
+| Run, Rerun and Run all from the dashboard | ✅ | ✅ | ⚠️ no Run all ¹ | ✅ |
+| Preserve & Rerun (compare a failed run with its rerun) | ✅ | ✅ | ✅ | ✅ |
+| Dashboard window opens by itself in live mode, and can be turned off | ⚠️ always opens ² | ✅ `openUi: false` | ⚠️ always opens ³ | ✅ `DEVTOOLS_OPEN=0` |
+| Turn off per-command screenshots (`captureScreenshots`) | ❌ ⁴ | ✅ | ❌ ⁴ | ❌ ⁴ |
+| Force the test browser headless (`headless`) | ❌ ⁵ | ⚠️ Chrome only ⁶ | ❌ ⁵ | ❌ ⁵ |
+| Safari kept frontmost so input lands (live mode) | ⚠️ once ⁷ | ✅ before each input | ⚠️ once ⁷ | ⚠️ once ⁷ |
+
+✅ tested: proven for that adapter by the cross-adapter conformance suite. Every other mark is declared.
+
+¹ The backend reruns a Nightwatch config through its framework path, which selects suites and tests but has no whole-run entry.
+
+² The service always opens the dashboard in live mode; `devtoolsCapabilities` picks its browser but cannot skip it.
+
+³ Nightwatch always opens the dashboard in live mode.
+
+⁴ No option: every command row takes a screenshot.
+
+⁵ No option: set headless in the browser capabilities, as for any run.
+
+⁶ The option adds `--headless=old` to `goog:chromeOptions` only, so other browsers stay headed; set their own headless capability.
+
+⁷ Brings Safari to the front once after the session starts (live mode, local macOS), so a click issued after another window took the front can still land nowhere.
+
+<!-- capability-matrix:readme-dashboard:end -->
+
 ### 📊 Real-Time Visualization
 - **Live Browser Preview**: View the application under test in a scaled iframe with automatic screenshot updates
 - **Actions Timeline**: Command-by-command execution log with timestamps and parameters
@@ -102,29 +141,68 @@ The dashboard runs in one of **two modes**, set per adapter via the shared `mode
 
 All three JavaScript adapters (`@wdio/devtools-service`, `@wdio/selenium-devtools`, `@wdio/nightwatch-devtools`) emit the **same normalized trace** through the shared `@wdio/devtools-core` capture library, so one archive format and one player serve every framework.
 
-The trace **format and the player are identical** across those three, but **capture completeness varies** — WebdriverIO is the most complete; Selenium and Nightwatch cover the core flow with some gaps (e.g. inline-Allure per-test artifacts, retry-aware retention, Cucumber step nesting, auto BiDi). Each adapter's README lists its specifics.
+The Python adapter (`selenium-devtools-py`) writes the same archive through the backend, so the trace **format and the player are identical** across all four, but **capture completeness varies**: WebdriverIO is the most complete, and the others cover the core flow with the gaps the table below names. Each adapter's README lists its specifics.
 
-**Trace-mode support by adapter:**
+**Support by adapter:**
 
-| Capability | WebdriverIO | Selenium | Nightwatch |
-|---|---|---|---|
-| Trace mode + `show-trace` player | ✅ | ✅ | ✅ |
-| DOM time-travel (mutation capture) | ✅ | ✅ ¹ | ✅ |
-| Dense filmstrip (`filmstrip`) | ✅ CDP push | ✅ CDP push | ⚠️ polling only |
-| Per-test granularity (`traceGranularity: 'test'`) | ✅ | ✅ | ⚠️ ² |
-| Retry-aware retention (`tracePolicy`) | ✅ | ✅ | ⚠️ ³ |
-| Cucumber step nesting | Scenario→Step ⁴ | ✅ full | Feature→Scenario ⁵ |
-| Inline Allure attach | ✅ | ✅ | ⚠️ produce-only ⁶ |
-| Assertion capture (`captureAssertions`) | ✅ | ✅ | ✅ |
-| Auto BiDi capture | ✅ auto | ✅ auto | ⚠️ opt-in ⁷ |
-
-¹ With BiDi, Selenium registers the page collector at document start, so every document anchors its own DOM as WebdriverIO's does. Without BiDi it falls back to injecting the collector after each navigation, and there a navigation's snapshot can lag the command that triggered it.
-² Nightwatch's Cucumber and exports-object interfaces get real per-test slicing; the BDD `describe/it` interface collapses to a single session-scoped slice keyed to the first test.
-³ Only `retain-on-failure` works; the other retry-aware policies degrade to it because Nightwatch's `--retries` re-runs a testcase internally without re-firing the per-test hooks.
-⁴ WebdriverIO does not yet carry feature-level ancestry, so its Cucumber nesting is Scenario→Step.
-⁵ Nightwatch does not yet stamp per-step nesting (Feature→Scenario only).
-⁶ Nightwatch has no live Allure attach API, so per-test `screenshot`/`video` are written to disk and listed in the manifest but not attached to an Allure test.
-⁷ Opt-in via `bidi: true` + `webSocketUrl: true` in capabilities.
+<!-- capability-matrix:readme-trace:start -->
+<!-- Generated by `pnpm docs:capabilities` from packages/conformance/src/capabilities.ts. Do not edit by hand. -->
+
+| Capability | WebdriverIO | Selenium | Nightwatch | Python |
+|---|---|---|---|---|
+| Trace mode + `show-trace` player | ✅ | ✅ | ✅ | ✅ |
+| DOM time-travel (mutation capture) | ✅ tested | ⚠️ with BiDi ¹ | ⚠️ with BiDi ¹ | ✅ tested |
+| Network requests (browser sessions) | ✅ tested | ⚠️ with BiDi ² | ⚠️ with BiDi ² | ✅ tested |
+| Screencast (`filmstrip` / video) | ✅ CDP push | ✅ CDP push | ⚠️ polling only ³ | ✅ CDP push |
+| Per-test granularity (`traceGranularity: 'test'`) | ✅ | ✅ | ⚠️ ⁴ | ✅ |
+| Retry-aware retention (`tracePolicy`) | ✅ | ✅ | ⚠️ `retain-on-failure` only ⁵ | ⚠️ `retain-on-failure` only ⁶ |
+| Per-test `screenshot` / `video`, attached to Allure | ✅ | ✅ | ⚠️ produce-only ⁷ | ❌ ⁸ |
+| Cucumber Feature→Scenario→Step nesting | ⚠️ Scenario→Step ⁹ | ✅ | ⚠️ Feature→Scenario ¹⁰ | ❌ ¹¹ |
+| Assertion capture (`captureAssertions`) | ✅ | ⚠️ ¹² | ✅ | ✅ |
+| BiDi console / network capture | ✅ auto | ✅ auto | ⚠️ opt-in ¹³ | ✅ auto |
+| A11y tab + pick-locator overlay (trace player) | ✅ | ✅ | ✅ | ✅ |
+| Transcript + Copy-for-LLM (trace player) | ✅ | ✅ | ✅ | ✅ |
+| Unpacked trace output (`traceFormat: 'ndjson-directory'`) | ✅ | ✅ | ✅ | ❌ zip only ¹⁴ |
+| Artifacts manifest for CI and reporters (`emitArtifactsManifest`) | ✅ | ✅ | ⚠️ opt-in ¹⁵ | ❌ ¹⁶ |
+| Network without BiDi (Safari, Appium mobile browsers) | ⚠️ ¹⁷ | ⚠️ ¹⁷ | ⚠️ ¹⁷ | ⚠️ ¹⁷ |
+
+✅ tested: proven for that adapter by the cross-adapter conformance suite. Every other mark is declared.
+
+¹ Without BiDi (Safari, Appium) the page collector is injected after each navigation, so a page the test leaves within tens of milliseconds is never captured.
+
+² Without BiDi a page the test leaves within tens of milliseconds is never drained, so its requests are not listed.
+
+³ Nightwatch exposes no stable CDP handle, so frames come from a `takeScreenshot` poll every 200 ms.
+
+⁴ The BDD `describe/it` interface fires the plugin's hooks once per module, so it collapses to one slice keyed to the first test; Cucumber and exports-object get real per-test slices.
+
+⁵ Nightwatch's `--retries` reruns a testcase internally without firing the per-test hooks again, so the other policies behave like `retain-on-failure`.
+
+⁶ Nothing the Python adapter sends carries an attempt number, so the retry-aware policies behave like `retain-on-failure`.
+
+⁷ Nightwatch has no live Allure attach API, so per-test screenshots and videos are written to disk and listed in the manifest but not attached.
+
+⁸ The Python adapter has no `screenshot` / `video` options and no Allure attachment yet.
+
+⁹ The service records no feature ancestry for a scenario, so the trace nests Scenario→Step.
+
+¹⁰ Nightwatch steps are not stamped as nested groups, so the trace nests Feature→Scenario.
+
+¹¹ The Python adapter stamps no Feature, Scenario or Step groups, so a pytest-bdd run nests as plain tests.
+
+¹² `node:assert` rows pass and fail; jest-style `expect()` and chai expose no pass hook, so only failing matchers appear there.
+
+¹³ Nightwatch turns it on with `bidi: true` plus `webSocketUrl: true` in the capabilities.
+
+¹⁴ The Python adapter has no option for it; the backend it asks always writes a zip.
+
+¹⁵ Nightwatch has no live Allure signal to turn the manifest on by itself, so it stays opt-in.
+
+¹⁶ The Python adapter writes no artifacts manifest.
+
+¹⁷ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari.
+
+<!-- capability-matrix:readme-trace:end -->
 
 In trace mode no DevTools UI window opens. At session end the adapter writes trace artifacts into a `test-results/` folder (created next to the resolved spec/config directory), suitable for offline replay, AI-agent diffing, or any consumer that prefers a portable artifact over a live UI.
 
@@ -133,6 +211,7 @@ In trace mode no DevTools UI window opens. At session end the adapter writes tra
 | **WebdriverIO** | `services: [['devtools', { mode: 'trace' }]]` |
 | **Selenium** | `DevTools.configure({ mode: 'trace' })` (before importing `selenium-webdriver`) |
 | **Nightwatch** | `globals: nightwatchDevtools({ mode: 'trace' })` |
+| **Python** | `pytest --devtools-trace`, or `devtools.enable(trace=True)` in a plain script |
 
 The trace artifact contains:
 - `trace.trace` — NDJSON `context-options` + `before`/`after` action events. When test hooks are available (Mocha's `it()` / Cucumber's `Scenario()`), each test becomes a `Tracing.tracingGroup` span — an open/close `before`/`after` pair with `method: "tracingGroup"` and `params.name` set to the test title. Child actions inside the group carry `parentId` pointing back to the group's `callId`, so timeline viewers render tests as labelled spans wrapping their commands.
@@ -216,7 +295,25 @@ The discriminator is the **browser name in the capabilities**, not the device: m
 
 A hybrid app is the case capabilities alone cannot answer, because the answer changes mid-run. Appium reports `NATIVE_APP` or a webview context, and anything that is not `NATIVE_APP` counts as a webview — the `WEBVIEW_` prefix is a convention, and a driver naming its webview otherwise would have its DOM capture skipped. Following the context costs no round trip: the switch command carries its destination in its own arguments.
 
-**Per-adapter:** all four adapters detect a native session and skip page-side capture on one. Only the **WebdriverIO** service additionally follows a hybrid app into its webview; Nightwatch, Selenium and the Python adapter answer from the startup capabilities, so a hybrid session's webview half is captured as if it were still native.
+**Per-adapter:**
+
+<!-- capability-matrix:readme-mobile:start -->
+<!-- Generated by `pnpm docs:capabilities` from packages/conformance/src/capabilities.ts. Do not edit by hand. -->
+
+| Capability | WebdriverIO | Selenium | Nightwatch | Python |
+|---|---|---|---|---|
+| Network without BiDi (Safari, Appium mobile browsers) | ⚠️ ¹ | ⚠️ ¹ | ⚠️ ¹ | ⚠️ ¹ |
+| Mobile browser sessions (Chrome on Android, Safari on iOS) | ✅ | ✅ | ✅ | ✅ |
+| iOS (XCUITest): native apps and Safari | ✅ | ✅ | ✅ | ✅ |
+| Native session detected, page-side capture skipped | ✅ | ✅ | ✅ | ✅ |
+| Follows a hybrid app into its webview | ✅ | ⚠️ ² | ⚠️ ² | ⚠️ ² |
+| Native app network through appium-interceptor (`appium:intercept`) | ✅ Android | ✅ Android | ✅ Android | ✅ Android |
+
+¹ Fetch and XHR calls carry headers and bodies; every other load comes from the page's Resource Timing buffer, with timing and size but no headers or body, and no status on Safari.
+
+² Decides from the startup capabilities, so a hybrid app's webview half is captured as if it were still native.
+
+<!-- capability-matrix:readme-mobile:end -->
 
 **In the dashboard**, a phone capture is framed as the device — a full-height device column with the action list and dock beside it — in both live mode and the trace player. The trace's `context-options` records the platform, model and OS version, read back so the player labels frames without guessing.
 

package.json

@@ -24,6 +24,7 @@
     "demo:python:mobile": "python3 examples/selenium-py/mobile/${DEVTOOLS_MOBILE_PLATFORM:-android}/*.py",
     "dev": "pnpm --parallel dev",
     "preview": "pnpm --parallel preview",
+    "docs:capabilities": "tsx packages/conformance/src/capability-matrix.ts",
     "test": "vitest run",
     "test:watch": "vitest",
     "test:coverage": "vitest run --coverage",