Repository navigation
test: add e2e coverage for the nx-plugin generators - #292
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
e4bb002 to
732e211
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
732e211 to
7c4004d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
7c4004d to
ba68ecc
Compare
|
Pushed a fix for e2e runtime. The first CI attempt ran for 25 minutes without finishing, so I cancelled it. Cause: Every Babel, Jest, SWC and Nx package pulled through a single-threaded Node proxy that also writes each one to disk. Warm locally, stone cold in CI. Fix: scope the local registry to the packages actually under test rather than raising the timeout.
Result, measured cold (
Exactly the two packages that need it; everything else resolves from npmjs at full speed. 3 tasks, 3 succeeded, and the working tree comes back clean with versions restored. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
ba68ecc to
c7ae7c4
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Adds an `@nx/plugin:e2e-project` based suite at `e2e/nx-plugin-e2e` that scaffolds a real Nx workspace, installs the plugin from the local registry and asserts against it. The generators were only covered by virtual-Tree unit tests, which cannot catch packaging, schema validation or plugin resolution problems. It found one on its first run: the plugin declared a peer dependency of `@nx/devkit@^16.3.2`, which drags in `@nx/devkit@16.10.0` and its `nx >= 15 <= 17` peer, so installing the plugin into an Nx 24 workspace fails with ERESOLVE. Widened to `>=16.3.2`. Replaces the hand-rolled local registry scripts with the canonical Nx versions. The old copy had drifted: - the stop handle was never assigned to `global.stopLocalRegistry`, so teardown was a no-op and verdaccio leaked after every run - it published via `nx run-many --targets publish`, and only `cli` has a `publish` target, so `nx-plugin` was never published at all Three guards on top of the canonical script, all needed here: - `gitPush: false`, because `nx.json` sets `release.git.push: true` for the real release pipeline and `releaseVersion` would otherwise push to origin during an e2e run - `registry` passed explicitly, because the `nx-release-publish` target default pins the public npm registry and wins over the environment `startLocalRegistry` sets, so publishing never reached verdaccio - the source manifests are restored in a `finally`, since `releaseVersion` writes the e2e version into them and would leave the tree dirty `releasePublish` resolves with a per-project status rather than throwing, so a publish failure is now surfaced directly instead of showing up later as an opaque ETARGET on install. `cli-e2e` now installs the CLI by the `e2e` dist-tag rather than a hardcoded `1.0.0`, so the published version stays an implementation detail. The `e2e` target is inferred through a scoped `@nx/jest/plugin` entry and gets its own CI job, since `run-many -t test` would not pick it up.
Three CI runs of `nx-plugin-e2e:e2e` hung after the build tasks and were killed by GitHub's 6 hour default, logging nothing past `nx run nx-plugin:build`. Nx buffers task output, so the failure left no trace to diagnose from. - `timeout-minutes: 20` on the job, so a hang costs minutes not hours - `--output-style=stream`, so output is not buffered away - `verbose: true` on `startLocalRegistry`, surfacing verdaccio's own startup output - a race around `startLocalRegistry`, which resolves only once verdaccio prints its URL and otherwise never settles Also stop the registry when globalSetup throws. Jest skips globalTeardown in that case, which strands verdaccio holding its port for every later run on the machine.
The suite was passing in CI in roughly 40 seconds, but the job then sat
until it was killed:
PASS nx-plugin-e2e src/nx-plugin.spec.ts (22.28 s)
Tests: 1 passed, 1 total
Jest did not exit one second after the test run has completed.
local registry exit 130
Teardown calls `childProcess.kill()` on the forked `nx` wrapper, but
verdaccio is its grandchild and survives on linux, holding the pipe open.
Nx buffers task output until a task completes, so a successful run that
never terminated produced no output at all - which is what made the
earlier runs look like a hang in the build step.
`forceExit` lets jest exit once teardown has signalled the registry.
The cli e2e installed the published CLI with `npm install -D`, which rewrote package.json and package-lock.json, and only undid it in teardown. Interrupting a run left the checkout carrying a `0.0.0-e2e` dependency and a churned lockfile. Install with `--no-save` instead: node_modules still gets the package, so the tests keep resolving a bare `user-flow` off the PATH, but both manifests stay byte-identical. Also pin npm to the workspace root rather than inheriting the runner's cwd, stop the registry when setup throws (vitest skips teardown in that case), and uninstall before the registry goes down instead of after.
The e2e scaffolded its target workspace with `create-nx-workspace` on every run. The manifest that generates floats - `typescript: ~6.0.3`, `@types/node: ^24.0.0`, `prettier: ^3.8.1`, `tslib: ^2.3.0` - and it ships no lockfile, so each run resolved a fresh tree of ~284 packages. An upstream release could turn the e2e red with nothing changed on our side, which is the kind of failure that reads as flake and gets re-run rather than investigated. Freeze that scaffold into a fixture instead, lockfile included, and `npm ci` from it. The tree is now pinned by integrity hash, and `ci` additionally refuses to run if the manifest and lockfile ever drift apart. Refresh it with `npm run e2e:update-workspace-fixture` after an Nx bump. The suite asserts the fixture's Nx matches the workspace's and fails with that command in the message, so a forgotten refresh cannot silently leave the plugin under test against a stale Nx. Side effect: the target runs in 29s rather than 1m14s, since copying a fixture and installing a lockfile beats downloading a template and resolving a tree.
`updateDependencies` built its dependency map with a bare `PLUGIN_NAME` as an object key, so the generator wrote a literal `"PLUGIN_NAME": "^0.0.0"` into the consumer's package.json - a name no registry can resolve, and a version that matched nothing even if it could. The workspace ended up without the plugin it had just been told to install. Use a computed key, and pin the plugin at the version that did the generating rather than a hardcoded `^0.0.0`. The assertion for this had been sitting commented out as `// TODO implement version test`; it is live now, alongside one that fails if the literal key ever comes back.
Eleven assertions required the CLI's stderr to be exactly empty. Node writes its own deprecation and experimental warnings to that same stream, so the suite passed or failed on which Node happened to be installed rather than on anything user-flow did - 9 of 23 tests failed on Node 26 with `DEP0205` while CI's Node 22 stayed green. Strip the runtime's own lines and assert on what is left, so the tests still catch the CLI writing to stderr without breaking on every new Node release. The suite now passes on Node 26 as well as on CI's Node 22.
The plugin e2e only checked that `npm ls` could see the package. It now runs both generators in the throwaway workspace and asserts what they produce: - init registers the plugin under its own name - the regression test for the literal `"PLUGIN_NAME"` dependency - pins it at the installed version, adds the CLI the executor shells out to, and sets the target generator's default - target writes the rc file, points the generated target at the plugin's executor, and stays resolvable through `nx show project`, which exercises the plugin's executors.json rather than just re-reading the json we wrote - target refuses to overwrite an existing target, and rejects a run that omits the required projectName Also ignore nested node_modules: vitest caches into `<project>/node_modules`, and the root-anchored pattern left a test run dirtying the tree. Not covered: running `nx run <project>:user-flow` for real. That would install puppeteer into the test workspace and drive Chrome, reintroducing exactly the network and version nondeterminism the fixture lockfile just removed.
27eb80c to
e7a5f6b
Compare
❗❗❗ report generated by this PR ❗❗❗☕ Order Coffee ☕Date/Time: 2026-10-24 19:27
|
☕ Order Coffee ☕Date/Time: 2026-10-24 19:27
|
Rebased on the Nx 24 upgrade (#291).
Why
The nx-plugin generators were only covered by virtual-
Treeunit tests (createTreeWithEmptyWorkspace), which call the generator factory directly. That left a blind spot around everything the Nx CLI does for real:generators.json/executors.json/schema.jsonsurvive the build'sassetsglobsschema.jsonvalidation, which only the CLI enforcesThere was a live bug sitting in that gap, and the e2e found it — see The bug this found below.
What
Adds
e2e/nx-plugin-e2e, generated with@nx/plugin:e2e-project. It installs the plugin from the local registry by dist-tag into a real Nx workspace and runs both generators against it.Local registry scripts
tools/scripts/{start,stop}-local-registry.tsare replaced with the canonical Nx versions. Nx flags the old copy itself when you run the generator:The drift cost two things:
global.stopLocalRegistry = await startLocalRegistry(…)releaseVersion+releasePublishnx run-many --targets publish— onlyclihas that target, sonx-pluginwas never publishedThe second row is what blocked plugin e2e entirely.
Guards on top of the canonical script
All found by running it:
gitPush: false.nx.jsonsetsrelease.git.push: truefor the real release pipeline, and the canonical script only overridesgitCommit/gitTag/stageChanges. The first run printedNX Pushing to git remote "origin"and only failed because the branch had no upstream. With tracking configured it would have pushed.finally.releaseVersionwrites0.0.0-e2eintopackages/*/package.jsonon disk, leaving the tree dirty.registryonreleasePublish.targetDefaults["nx-release-publish"]pins the public registry, and that target option beats thenpm_config_registryenv varstartLocalRegistrysets. Without it, publishing tried to authenticate against npmjs and the failure surfaced much later as an opaqueETARGETon install.releasePublishresolves with a per-project status map rather than throwing, so a failed publish otherwise limped on.startLocalRegistryresolves only once verdaccio prints its URL and never rejects if that line doesn't arrive.Wiring on Nx 24
@nx/jest:jestno longer exists as an executor, so thee2etarget is inferred via a scoped@nx/jest/pluginentry withtargetName: "e2e", matching howpackages/nx-plugingets itstesttarget@nx/jest/pluginentry withtargetName: "jest:test"; dropped it, since the repo already scopes jest inference deliberatelye2e/nx-plugin-e2e/**/*added to the@nx/eslint/plugininclude listjest.config.cts, CJS, matching the rest of the workspace after the migratione2eCI job withtimeout-minutes: 20, sincerun-many -t testwould not pick the target upThe bug this found
updateDependenciesbuilt its dependency map with a barePLUGIN_NAMEas an object key:So
nx g @push-based/user-flow-nx-plugin:initwrote a literal"PLUGIN_NAME": "^0.0.0"into the consumer'spackage.json— a name no registry can resolve, at a version that matched nothing even if it could. A workspace ended up without the plugin it had just been told to install.Fixed with a computed key, pinned at the version that did the generating rather than a hardcoded
^0.0.0. The assertion for it had been sitting commented out as// TODO implement version test; it is live now, next to one that fails if the literal key returns.Determinism
Two problems made the suites pass or fail for reasons unrelated to the code.
The target workspace resolved a fresh dependency tree on every run.
create-nx-workspacegenerates a manifest that floats —typescript: ~6.0.3,@types/node: ^24.0.0,prettier: ^3.8.1,tslib: ^2.3.0— and ships no lockfile, so an upstream release could turn the e2e red with nothing changed here. That scaffold is now frozen intoe2e/nx-plugin-e2e/fixtures/workspace/, lockfile included, and the suitenpm cis from it. The tree is pinned by integrity hash, andciadditionally refuses to run if the manifest and lockfile drift apart.Refresh it with
npm run e2e:update-workspace-fixtureafter an Nx bump. The suite asserts the fixture's Nx matches the workspace's and fails with that command in the message, so a forgotten refresh can't silently leave the plugin under test against a stale Nx.Side effect: the target runs in ~50s rather than 1m14s, since copying a fixture and installing a lockfile beats downloading a template and resolving a tree.
cli-e2easserted on Node's own output. Eleven assertions required the CLI's stderr to be exactly empty, but Node writes its deprecation and experimental warnings to that same stream — 9 of 23 tests failed on Node 26 withDEP0205while CI's Node 22 stayed green. The runtime's own lines are now stripped before the comparison, so the tests still catch the CLI writing to stderr without breaking on every new Node release.Checkout hygiene
cli-e2einstalled the published CLI withnpm install -D, rewritingpackage.jsonandpackage-lock.jsonand only undoing it in teardown — interrupting a run left the checkout carrying a0.0.0-e2edependency and a churned lockfile. It installs with--no-savenow:node_modulesstill gets the package, so the tests keep resolving a bareuser-flowoff the PATH, but both manifests stay byte-identical..gitignorenow ignores nestednode_modules; vitest caches into<project>/node_modulesand the root-anchored pattern left a test run dirtying the tree.mkdtempoutside the repo, unique per call, so concurrent worktrees don't collide.What the e2e asserts
init— registers the plugin under its own name (the regression test for the literal"PLUGIN_NAME"), pins it at the installed version, adds the CLI the executor shells out to, and sets the target generator's default target name innx.json.target— writes the rc file, points the generated target at the plugin's executor, and stays resolvable throughnx show project, which exercises the plugin'sexecutors.jsonrather than just re-reading the json we wrote. It also refuses to overwrite an existing target and rejects a run that omits the requiredprojectName.Not covered: running
nx run <project>:user-flowfor real. That would install puppeteer into the test workspace and drive Chrome, reintroducing exactly the network and version nondeterminism the fixture lockfile removes.Verification
Green locally and on CI:
cliunitnx-pluginunitcli-e2enx-plugin-e2ecli-e2ewas 9 red locally before the stderr change. The plugin e2e installs the same 284 packages on both machines, which is the determinism actually showing up rather than being asserted.Two failures worth recording, since both were invisible rather than loud:
Jest did not exit one second after the test run has completed.Teardown'schildProcess.kill()reaches the forkednxwrapper, but verdaccio is its grandchild and survives on Linux holding the pipe. Nx buffers task output until completion, so a passing-but-unterminated task logged nothing at all. Fixed withforceExit, plus--output-style=streamand a job timeout so the next one reports instead of burning six hours.startLocalRegistrysetsnpm_config_registryfor the whole process, so scaffolding dragged ~220MB across 159 scopes through the local proxy. Only@push-based/*needs it; the rest is sent back to the public registry.