Repository navigation
feat: support running TypeScript benchmarks with tsx - #100
adriencaccia wants to merge 9 commits into
Conversation
When a benchmark runs with tsx under CodSpeed, the introspected flags now preload a loader ahead of tsx that patches the esbuild instance tsx uses: - `keepNames` and `minifyWhitespace` are turned off for tsx's own transforms. `keepNames` wraps functions in `__name()` calls that run inside the measured code (+30% on closure-heavy benchmarks) and hide arrow function names from V8, so flamegraphs showed `anonymous function at line 2` instead of the name. - esbuild's service processes start without `--ping`. With it, they ping their host thread every second, and that thread's work was counted in random simulation measurements. The loader is a `--require` preload, so it runs before tsx's own preloads. When tsx cannot be patched, it warns and leaves tsx's defaults. Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
…mark When the benchmark file calls `run()` while an ES module loader still has promise jobs queued, those jobs run between the awaits of the first measurements and add up to 250k instructions to a random early benchmark. Draining one event loop turn first makes these runs deterministic. Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
esbuild-register is no longer maintained. Use tsx, which the ES module examples already use, and drop esbuild-register. Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
Run the simple ES module example with a top-level `await bench.run()` and add an async benchmark, the shape users write when running TypeScript benchmarks with tsx. Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
`tsx bench.ts` runs the script in a child node process, which gets neither the node flags CodSpeed adds to the CLI's process nor the tsx loader. When the plugin loads in such a process without the flags, rerun it with them, in place with `process.execve` when available, so no parent process stays alive during the measurements. A process reruns at most once, and never from a worker thread or with an IPC channel to its parent. The runner splits the introspected flags on whitespace, so flags holding a path with spaces are left out of them; under tsx, the rerun adds them back. Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
Refs COD-3900 Co-Authored-By: Claude <noreply@anthropic.com>
Check that tsx transforms without keepNames and minifyWhitespace, that esbuild's service starts without `--ping`, and that a script run with the `tsx` CLI under CodSpeed reruns once with the node flags it needs. Refs COD-3900 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsx registers its module hooks off-thread before Node 22.22.3 and 24.11.1, and Node only has `module.registerHooks` from 22.15, so the latest 22, 24 and 26 releases never exercise those loader paths. Refs COD-3900 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merging this PR will regress 4 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Memory | test_recursive_cached_fibo_10 |
784 B | 1,440 B | -45.56% |
| ❌ | Simulation | recursive fibo 15 |
350.7 µs | 635.6 µs | -44.83% |
| ❌ | Memory | test sync baz 10 |
784 B | 1,312 B | -40.24% |
| ❌ | WallTime | switch 2 |
84 ns | 108 ns | -22.22% |
| ⚡ | Memory | switch 2 |
13,384 B | 656 B | ×20 |
| ⚡ | WallTime | test_recursive_cached_fibo_10 |
2,544 ns | 612 ns | ×4.2 |
| ⚡ | WallTime | test_recursive_cached_fibo_20 |
4 µs | 1.7 µs | ×2.4 |
| ⚡ | WallTime | test_recursive_cached_fibo_30 |
4.7 µs | 2.2 µs | ×2.1 |
| ⚡ | Simulation | test_recursive_cached_fibo_10 |
85.3 µs | 45.7 µs | +86.71% |
| ⚡ | Simulation | recursive fibo 15 with hooks |
644.1 µs | 362 µs | +77.94% |
| ⚡ | Simulation | recursive fibo 15 with hooks |
644.2 µs | 362.2 µs | +77.89% |
| ⚡ |
Simulation | wait 500ms |
47.9 ms | 27.8 ms | +72.52% |
| ⚡ |
Simulation | wait 500ms |
47.1 ms | 28.1 ms | +67.77% |
| ⚡ |
Simulation | wait 1sec |
85.1 ms | 56.2 ms | +51.46% |
| ⚡ |
Simulation | wait 1sec |
83.7 ms | 56.9 ms | +47.07% |
| ⚡ | Simulation | test_recursive_cached_fibo_20 |
81.9 µs | 56.7 µs | +44.52% |
| ⚡ | Simulation | test_recursive_cached_fibo_30 |
94.8 µs | 69.7 µs | +36.06% |
| ⚡ | Simulation | test_recursive_cached_fibo_20 |
109.8 µs | 81.8 µs | +34.23% |
| ⚡ ℹ️ | Simulation | test_recursive_cached_fibo_10 |
112 µs | 83.5 µs | +34.22% |
| ⚡ | Simulation | test_recursive_cached_fibo_30 |
117.6 µs | 89.6 µs | +31.25% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing cod-3900-use-tsx-instead-of-esbuild-register-for-nodejs-typescript (9fab634) with main (d15b97d)
Footnotes
-
57 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Make tsx safe to benchmark with CodSpeed, so the docs can recommend it instead of the unmaintained esbuild-register, which cannot run a top-level
await. Under CodSpeed, a preloaded loader turns off the esbuild options tsx forces (keepNamesran__name()calls inside the measured code, up to +80% on closure-heavy benchmarks, and hid arrow function names in flamegraphs) and the--pingof esbuild's service, whose handling every second was counted in random simulation measurements; scripts started with thetsxCLI rerun themselves with the runner's node flags, and tinybench now lets promise jobs left by module loading run before its first measurement. On Node 22.13 to 26.8, simulation instruction counts, walltime and memory results match esbuild-register.Refs COD-3900