Skip to content

test: add e2e coverage for the nx-plugin generators - #292

Merged
ChristopherPHolder merged 8 commits into
mainfrom
test/nx-plugin-e2e
Oct 24, 2026
Merged

ChristopherPHolder merged 8 commits into
mainfrom
test/nx-plugin-e2e

Conversation

@ChristopherPHolder

@ChristopherPHolder ChristopherPHolder commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Rebased on the Nx 24 upgrade (#291).

Why

The nx-plugin generators were only covered by virtual-Tree unit tests (createTreeWithEmptyWorkspace), which call the generator factory directly. That left a blind spot around everything the Nx CLI does for real:

  • whether generators.json / executors.json / schema.json survive the build's assets globs
  • schema.json validation, which only the CLI enforces
  • whether the plugin resolves by package name in a consuming workspace
  • whether a generated target actually runs afterwards

There was a live bug sitting in that gap, and the e2e found it — see The bug this found below.

What

Adds e2e/nx-plugin-e2e, generated with @nx/plugin:e2e-project. It installs the plugin from the local registry by dist-tag into a real Nx workspace and runs both generators against it.

Local registry scripts

tools/scripts/{start,stop}-local-registry.ts are replaced with the canonical Nx versions. Nx flags the old copy itself when you run the generator:

Your start-local-registry.ts script may be outdated. To ensure that newly generated packages are published appropriately when running end to end tests, update this script to use Nx Release.

The drift cost two things:

canonical old copy
teardown global.stopLocalRegistry = await startLocalRegistry(…) return value discarded → teardown was a no-op, verdaccio leaked after every run
publishing releaseVersion + releasePublish nx run-many --targets publish — only cli has that target, so nx-plugin was never published

The second row is what blocked plugin e2e entirely.

Guards on top of the canonical script

All found by running it:

  1. gitPush: false. nx.json sets release.git.push: true for the real release pipeline, and the canonical script only overrides gitCommit/gitTag/stageChanges. The first run printed NX Pushing to git remote "origin" and only failed because the branch had no upstream. With tracking configured it would have pushed.
  2. Manifest restore in a finally. releaseVersion writes 0.0.0-e2e into packages/*/package.json on disk, leaving the tree dirty.
  3. Explicit registry on releasePublish. targetDefaults["nx-release-publish"] pins the public registry, and that target option beats the npm_config_registry env var startLocalRegistry sets. Without it, publishing tried to authenticate against npmjs and the failure surfaced much later as an opaque ETARGET on install.
  4. Fail fast on publish errors. releasePublish resolves with a per-project status map rather than throwing, so a failed publish otherwise limped on.
  5. A timeout on registry startup. startLocalRegistry resolves only once verdaccio prints its URL and never rejects if that line doesn't arrive.

Wiring on Nx 24

  • @nx/jest:jest no longer exists as an executor, so the e2e target is inferred via a scoped @nx/jest/plugin entry with targetName: "e2e", matching how packages/nx-plugin gets its test target
  • the generator also added a workspace-wide @nx/jest/plugin entry with targetName: "jest:test"; dropped it, since the repo already scopes jest inference deliberately
  • e2e/nx-plugin-e2e/**/* added to the @nx/eslint/plugin include list
  • jest.config.cts, CJS, matching the rest of the workspace after the migration
  • new e2e CI job with timeout-minutes: 20, since run-many -t test would not pick the target up

The bug this found

updateDependencies built its dependency map with a bare PLUGIN_NAME as an object key:

const devDeps = {
  '@push-based/user-flow': '^0.19.0',
  PLUGIN_NAME: '^0.0.0',
  '@nx/devkit': '^16.0.0'
};

So nx g @push-based/user-flow-nx-plugin:init wrote a literal "PLUGIN_NAME": "^0.0.0" into the consumer's package.json — a name no registry can resolve, at a version that matched nothing even if it could. A workspace ended up without the plugin it had just been told to install.

Fixed with a computed key, pinned at the version that did the generating rather than a hardcoded ^0.0.0. The assertion for it had been sitting commented out as // TODO implement version test; it is live now, next to one that fails if the literal key returns.

Determinism

Two problems made the suites pass or fail for reasons unrelated to the code.

The target workspace resolved a fresh dependency tree on every run. create-nx-workspace generates a manifest that floats — typescript: ~6.0.3, @types/node: ^24.0.0, prettier: ^3.8.1, tslib: ^2.3.0 — and ships no lockfile, so an upstream release could turn the e2e red with nothing changed here. That scaffold is now frozen into e2e/nx-plugin-e2e/fixtures/workspace/, lockfile included, and the suite npm cis from it. The tree is pinned by integrity hash, and ci additionally refuses to run if the manifest and lockfile drift apart.

Refresh it with npm run e2e:update-workspace-fixture after an Nx bump. The suite asserts the fixture's Nx matches the workspace's and fails with that command in the message, so a forgotten refresh can't silently leave the plugin under test against a stale Nx.

Side effect: the target runs in ~50s rather than 1m14s, since copying a fixture and installing a lockfile beats downloading a template and resolving a tree.

cli-e2e asserted on Node's own output. Eleven assertions required the CLI's stderr to be exactly empty, but Node writes its deprecation and experimental warnings to that same stream — 9 of 23 tests failed on Node 26 with DEP0205 while CI's Node 22 stayed green. The runtime's own lines are now stripped before the comparison, so the tests still catch the CLI writing to stderr without breaking on every new Node release.

Checkout hygiene

  • cli-e2e installed the published CLI with npm install -D, rewriting package.json and package-lock.json and only undoing it in teardown — interrupting a run left the checkout carrying a 0.0.0-e2e dependency and a churned lockfile. It installs with --no-save now: node_modules still gets the package, so the tests keep resolving a bare user-flow off the PATH, but both manifests stay byte-identical.
  • npm is pinned to the workspace root rather than inheriting the runner's cwd, the registry is stopped when setup throws (vitest skips teardown in that case), and uninstall happens before the registry goes down instead of after.
  • .gitignore now ignores nested node_modules; vitest caches into <project>/node_modules and the root-anchored pattern left a test run dirtying the tree.
  • The test workspace is created under mkdtemp outside the repo, unique per call, so concurrent worktrees don't collide.

What the e2e asserts

init — registers the plugin under its own name (the regression test for the literal "PLUGIN_NAME"), pins it at the installed version, adds the CLI the executor shells out to, and sets the target generator's default target name in nx.json.

target — writes the rc file, points the generated target at the plugin's executor, and stays resolvable through nx show project, which exercises the plugin's executors.json rather than just re-reading the json we wrote. It also refuses to overwrite an existing target and rejects a run that omits the required projectName.

Not covered: running nx run <project>:user-flow for real. That would install puppeteer into the test workspace and drive Chrome, reintroducing exactly the network and version nondeterminism the fixture lockfile removes.

Verification

Green locally and on CI:

suite local (Node 26) CI (Node 22)
cli unit 74/74 pass
nx-plugin unit 13/13 pass
cli-e2e 23/23 pass
nx-plugin-e2e 10/10 pass

cli-e2e was 9 red locally before the stderr change. The plugin e2e installs the same 284 packages on both machines, which is the determinism actually showing up rather than being asserted.

Two failures worth recording, since both were invisible rather than loud:

  • A 6-hour CI job. The suite passed in ~40s, then Jest did not exit one second after the test run has completed. Teardown's childProcess.kill() reaches the forked nx wrapper, but verdaccio is its grandchild and survives on Linux holding the pipe. Nx buffers task output until completion, so a passing-but-unterminated task logged nothing at all. Fixed with forceExit, plus --output-style=stream and a job timeout so the next one reports instead of burning six hours.
  • Everything proxied through verdaccio. startLocalRegistry sets npm_config_registry for the whole process, so scaffolding dragged ~220MB across 159 scopes through the local proxy. Only @push-based/* needs it; the rest is sent back to the public registry.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ChristopherPHolder
ChristopherPHolder force-pushed the test/nx-plugin-e2e branch 2 times, most recently from e4bb002 to 732e211 Compare October 23, 2026 20:27
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ChristopherPHolder

Copy link
Copy Markdown
Collaborator Author

Pushed a fix for e2e runtime. The first CI attempt ran for 25 minutes without finishing, so I cancelled it.

Cause: startLocalRegistry sets npm_config_registry for the whole process, so create-nx-workspace scaffolded the test workspace's entire dependency tree through verdaccio. Measured on a local run:

verdaccio storage: 224M across 159 scopes

Every Babel, Jest, SWC and Nx package pulled through a single-threaded Node proxy that also writes each one to disk. Warm locally, stone cold in CI.

Fix: scope the local registry to the packages actually under test rather than raising the timeout.

  • create-nx-workspace and the plugin install run with npm_config_registry pointed back at the public registry
  • the generated workspace gets an .npmrc with @push-based:registry=<verdaccio> and its auth token

Result, measured cold (rm -rf tmp/local-registry first):

before after
verdaccio storage 224 MB 516 KB
scopes proxied 159 1
wall clock (cold, local) — 39s
$ ls tmp/local-registry/storage/@push-based/
user-flow  user-flow-nx-plugin

Exactly the two packages that need it; everything else resolves from npmjs at full speed. 3 tasks, 3 succeeded, and the working tree comes back clean with versions restored.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

Adds an `@nx/plugin:e2e-project` based suite at `e2e/nx-plugin-e2e` that
scaffolds a real Nx workspace, installs the plugin from the local registry
and asserts against it. The generators were only covered by virtual-Tree
unit tests, which cannot catch packaging, schema validation or plugin
resolution problems.

It found one on its first run: the plugin declared a peer dependency of
`@nx/devkit@^16.3.2`, which drags in `@nx/devkit@16.10.0` and its
`nx >= 15 <= 17` peer, so installing the plugin into an Nx 24 workspace
fails with ERESOLVE. Widened to `>=16.3.2`.

Replaces the hand-rolled local registry scripts with the canonical Nx
versions. The old copy had drifted:

- the stop handle was never assigned to `global.stopLocalRegistry`, so
  teardown was a no-op and verdaccio leaked after every run
- it published via `nx run-many --targets publish`, and only `cli` has a
  `publish` target, so `nx-plugin` was never published at all

Three guards on top of the canonical script, all needed here:

- `gitPush: false`, because `nx.json` sets `release.git.push: true` for
  the real release pipeline and `releaseVersion` would otherwise push to
  origin during an e2e run
- `registry` passed explicitly, because the `nx-release-publish` target
  default pins the public npm registry and wins over the environment
  `startLocalRegistry` sets, so publishing never reached verdaccio
- the source manifests are restored in a `finally`, since `releaseVersion`
  writes the e2e version into them and would leave the tree dirty

`releasePublish` resolves with a per-project status rather than throwing,
so a publish failure is now surfaced directly instead of showing up later
as an opaque ETARGET on install.

`cli-e2e` now installs the CLI by the `e2e` dist-tag rather than a
hardcoded `1.0.0`, so the published version stays an implementation detail.

The `e2e` target is inferred through a scoped `@nx/jest/plugin` entry and
gets its own CI job, since `run-many -t test` would not pick it up.
Three CI runs of `nx-plugin-e2e:e2e` hung after the build tasks and were
killed by GitHub's 6 hour default, logging nothing past `nx run
nx-plugin:build`. Nx buffers task output, so the failure left no trace to
diagnose from.

- `timeout-minutes: 20` on the job, so a hang costs minutes not hours
- `--output-style=stream`, so output is not buffered away
- `verbose: true` on `startLocalRegistry`, surfacing verdaccio's own startup
  output
- a race around `startLocalRegistry`, which resolves only once verdaccio
  prints its URL and otherwise never settles

Also stop the registry when globalSetup throws. Jest skips globalTeardown in
that case, which strands verdaccio holding its port for every later run on
the machine.
The suite was passing in CI in roughly 40 seconds, but the job then sat
until it was killed:

    PASS   nx-plugin-e2e  src/nx-plugin.spec.ts (22.28 s)
    Tests:       1 passed, 1 total
    Jest did not exit one second after the test run has completed.
    local registry exit 130

Teardown calls `childProcess.kill()` on the forked `nx` wrapper, but
verdaccio is its grandchild and survives on linux, holding the pipe open.
Nx buffers task output until a task completes, so a successful run that
never terminated produced no output at all - which is what made the
earlier runs look like a hang in the build step.

`forceExit` lets jest exit once teardown has signalled the registry.
The cli e2e installed the published CLI with `npm install -D`, which rewrote
package.json and package-lock.json, and only undid it in teardown. Interrupting
a run left the checkout carrying a `0.0.0-e2e` dependency and a churned
lockfile.

Install with `--no-save` instead: node_modules still gets the package, so the
tests keep resolving a bare `user-flow` off the PATH, but both manifests stay
byte-identical.

Also pin npm to the workspace root rather than inheriting the runner's cwd, stop
the registry when setup throws (vitest skips teardown in that case), and
uninstall before the registry goes down instead of after.
The e2e scaffolded its target workspace with `create-nx-workspace` on every
run. The manifest that generates floats - `typescript: ~6.0.3`,
`@types/node: ^24.0.0`, `prettier: ^3.8.1`, `tslib: ^2.3.0` - and it ships no
lockfile, so each run resolved a fresh tree of ~284 packages. An upstream
release could turn the e2e red with nothing changed on our side, which is the
kind of failure that reads as flake and gets re-run rather than investigated.

Freeze that scaffold into a fixture instead, lockfile included, and `npm ci`
from it. The tree is now pinned by integrity hash, and `ci` additionally
refuses to run if the manifest and lockfile ever drift apart.

Refresh it with `npm run e2e:update-workspace-fixture` after an Nx bump. The
suite asserts the fixture's Nx matches the workspace's and fails with that
command in the message, so a forgotten refresh cannot silently leave the plugin
under test against a stale Nx.

Side effect: the target runs in 29s rather than 1m14s, since copying a fixture
and installing a lockfile beats downloading a template and resolving a tree.
`updateDependencies` built its dependency map with a bare `PLUGIN_NAME` as an
object key, so the generator wrote a literal `"PLUGIN_NAME": "^0.0.0"` into the
consumer's package.json - a name no registry can resolve, and a version that
matched nothing even if it could. The workspace ended up without the plugin it
had just been told to install.

Use a computed key, and pin the plugin at the version that did the generating
rather than a hardcoded `^0.0.0`.

The assertion for this had been sitting commented out as `// TODO implement
version test`; it is live now, alongside one that fails if the literal key ever
comes back.
Eleven assertions required the CLI's stderr to be exactly empty. Node writes its
own deprecation and experimental warnings to that same stream, so the suite
passed or failed on which Node happened to be installed rather than on anything
user-flow did - 9 of 23 tests failed on Node 26 with `DEP0205` while CI's Node
22 stayed green.

Strip the runtime's own lines and assert on what is left, so the tests still
catch the CLI writing to stderr without breaking on every new Node release.

The suite now passes on Node 26 as well as on CI's Node 22.
The plugin e2e only checked that `npm ls` could see the package. It now runs
both generators in the throwaway workspace and asserts what they produce:

- init registers the plugin under its own name - the regression test for the
  literal `"PLUGIN_NAME"` dependency - pins it at the installed version, adds
  the CLI the executor shells out to, and sets the target generator's default
- target writes the rc file, points the generated target at the plugin's
  executor, and stays resolvable through `nx show project`, which exercises the
  plugin's executors.json rather than just re-reading the json we wrote
- target refuses to overwrite an existing target, and rejects a run that omits
  the required projectName

Also ignore nested node_modules: vitest caches into `<project>/node_modules`,
and the root-anchored pattern left a test run dirtying the tree.

Not covered: running `nx run <project>:user-flow` for real. That would install
puppeteer into the test workspace and drive Chrome, reintroducing exactly the
network and version nondeterminism the fixture lockfile just removed.
@github-actions

Copy link
Copy Markdown

❗❗❗ report generated by this PR ❗❗❗

☕ Order Coffee ☕

Date/Time: 2026-10-24 19:27

Step Name Gather Mode Performance Accessibility Best Practices Seo Agentic Browsing
🧭 Navigate to coffee cart navigation 92 93 100 83 50
☕ Select coffee timespan 17/17 - 8/8 - -
✔ Coffee selected snapshot Ø 1/1 16/18 4/4 3/5 1/3
🛒 Checkout order timespan 19/19 - 8/8 - -
🧾 Order checked out snapshot Ø 1/1 19/20 4/4 3/5 1/3
💌 Submit order timespan 18/18 - 8/8 - -
📧 Order submitted snapshot Ø 1/1 20/22 4/4 3/5 1/3
🧭 Navigate to github navigation 98 92 100 83 50

@github-actions

Copy link
Copy Markdown

☕ Order Coffee ☕

Date/Time: 2026-10-24 19:27

Step Name Gather Mode Performance Accessibility Best Practices Seo Agentic Browsing
🧭 Navigate to coffee cart navigation 93 93 100 83 50
☕ Select coffee timespan 17/17 - 8/8 - -
✔ Coffee selected snapshot Ø 1/1 16/18 4/4 3/5 1/3
🛒 Checkout order timespan 19/19 - 8/8 - -
🧾 Order checked out snapshot Ø 1/1 19/20 4/4 3/5 1/3
💌 Submit order timespan 19/19 - 8/8 - -
📧 Order submitted snapshot Ø 1/1 20/22 4/4 3/5 1/3
🧭 Navigate to github navigation 96 92 100 83 50

@ChristopherPHolder
ChristopherPHolder merged commit 241c619 into main Oct 24, 2026
4 checks passed
@ChristopherPHolder
ChristopherPHolder deleted the test/nx-plugin-e2e branch October 24, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant