Repository navigation
Measure API client allocations and client/server memory in separate passes - #164
Draft
Andrew Branch (andrewbranch) wants to merge 3 commits into
Draft
Andrew Branch (andrewbranch) wants to merge 3 commits into
Andrew Branch (andrewbranch) wants to merge 3 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2e98ee2-8ada-4ded-92fb-05508196941e
Andrew Branch (andrewbranch)
requested a review
from Jake Bailey (jakebailey)
as a code owner
October 20, 2026 19:08
Andrew Branch (andrewbranch)
marked this pull request as draft
October 20, 2026 19:23
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2e98ee2-8ada-4ded-92fb-05508196941e
There was a problem hiding this comment.
🔵 Needs a closer look
The unpublished compiler prerequisite and unverified Linux sandbox execution and peak-RSS behavior require human validation.
No open issues 🎉
What changed in this PR
Adds opt-in API allocation and memory measurements through separate profiling passes, using the existing benchmark and reporting model.
Changes:
- Separates timing and memory runs, validates telemetry, and aggregates process/window measurements.
- Adds memory metrics and rejects comparisons with inconsistent memory coverage.
- Wires the opt-in through CI and the sandbox, with tests and documentation.
| File | Description |
|---|---|
| ts-perf/packages/commands/src/benchmark/runApiScenario.ts | Runs isolated memory passes after timing. |
| ts-perf/packages/commands/src/benchmark/runApiScenario.test.ts | Tests pass isolation and failures. |
| ts-perf/packages/commands/src/benchmark/index.ts | Adds the memory option type. |
| ts-perf/packages/commands/src/benchmark/apiMemory.ts | Validates and aggregates memory dumps. |
| ts-perf/packages/commands/src/benchmark/apiMemory.test.ts | Tests aggregation and report integration. |
| ts-perf/packages/commands/src/benchmark/api.ts | Exposes --apiMemory. |
| ts-perf/packages/api/src/model/sample.ts | Defines memory metrics. |
| ts-perf/packages/api/src/model/measurementComparison.ts | Guards against asymmetric memory comparisons. |
| scripts/src/setupPipeline.ts | Parses opt-in and accounts for extra runs. |
| scripts/src/runTsPerf.ts | Forwards opt-in to saved API runs. |
| scripts/src/__tests__/utils.test.ts | Tests conditional option forwarding. |
| scripts/src/__tests__/setupPipeline.test.ts | Tests defaults, restrictions, and cost accounting. |
| scripts/src/__tests__/scenarioSetup.test.ts | Supplies the sandbox memory variable. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/vscode/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/vscode_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/tsserver-only/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/tsserver-only_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/tsc-only/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/tsc-only_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/this is not a preset/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/this is not a preset_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/startup-only/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/startup-only_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/startup-only hosts=node@22.8.0/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/startup-only hosts=node@22.8.0_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/regular/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/regular_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/public/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/public_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/predictable=true/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/predictable=true_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/hosts=bun@1.1.3,vscode@1.88.1/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/hosts=bun@1.1.3,vscode@1.88.1_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/full/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/full_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/faster/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/faster_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/faster predictable host=node@18.5.1/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/faster predictable host=node@18.5.1_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/faster commits=release-5.3...release-5.4/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/faster commits=release-5.3...release-5.4_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/default/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/default_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/bun/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/bun_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/baseline/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/baseline_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/src/__tests__/__file_snapshots__/setupPipeline/api_tsgo/outputVariables.snap | Records memory-disabled default. |
| scripts/runBenchmarkSandbox.sh | Requires and forwards the memory variable. |
| README.md | Documents usage, metrics, and limitations. |
| build/benchmark.yml | Propagates memory configuration between jobs. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2e98ee2-8ada-4ded-92fb-05508196941e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Copilot slop:
Important
Source prerequisite is not published.
--apiMemoryrequires accompanying TypeScript compiler/API changes implementingTS_API_MEMORY_DIR, collected-allocation V8 sampling, the nativegetServerMemoryrequest, and the reviewed Linux clientVmHWMpeak-RSS correction. Those changes remain local, not included in this PR or an upstream prerequisite PR. Timing-only API benchmarks continue working without them; requesting memory on an older candidate fails explicitly. Integration used a patched source tree based on28534d7609062699a68b1b66e2367f8d95dbf290, not an unmodified released compiler.Adds client allocations and client/server heap memory using the existing scalar benchmark/report model. Server peak RSS has been removed at the user's request, including its producer wire field and OS-specific Go helpers. There is no replacement server RSS mechanism.
ts-perf benchmark api --apiMemory, or CI presetapi apiMemory, completes ordinary timing iterations with memory profiling disabled, then repeats the identical workload/candidate/installed Node/bundled npm in separate memory-only iterations. Memory runs have no additional discarded warmups: each starts fresh processes and the timing pass has already warmed OS caches. Both commits receive the same opt-in; asymmetric memory/timing-only report comparisons fail.TSPERF_API_MEMORY, required at the sandbox boundary and forwarded only for saved API runs, not load-only comparisons. The runner consumes the producer's byte estimate without requiring a particular sampling interval.includeObjectsCollectedByMajorGCandincludeObjectsCollectedByMinorGC. Both heap snapshots include uncollected garbage. GoTotalAllocis exact cumulative Go heap allocated bytes. Client RSS is lifetime high-water, not live heap./proc/self/statusVmHWMfor the current executable image, notgetrusage's inherited pre-exec launcher high-water mark. macOS retains its existing client rusage measurement.{pid, heapAllocBytes, totalAllocBytes}. Retained memory metrics are client sampled JS allocations, client heap, client peak RSS, server Go TotalAlloc, and server heap, plus client/server process counts. Missing/unsupported retained counters fail the requested memory pass. No runner preload, compiler source rewriting, forced GC, new report format, or availability/provenance protocol.JS allocation/heap metrics cover the main V8 isolate, not worker isolates/native allocations; client RSS covers the whole client process. Profiling windows cover overlapping API-client lifetimes, not all command startup/shutdown. Profiling overhead and sampling noise remain. Worker-thread collection fails explicitly. Requested heap snapshots and process counts are retained; the comparison guard also protects local CLI and saved-report comparisons outside CI.
Validation
npm run generate:metaModel, floating-rulenpm run lint, and recommended-rulenpm run lint. Each ran as separate timing/memory passes twice (same patched compiler, labeled baseline/candidate to exercise report plumbing, not a performance comparison between revisions). Every memory run reported exactly one API client and one server despite npm/wrappers. Production packages came from the producer's standard packaging task and existingpreserveApiArtifact; telemetry JS and native binary bytes were unchanged during staging.global.gc,--expose-gc, or forced-GC inspector request; heap limits were used only for this stress test, not benchmark workloads.main: macOSchmod --, Bash3.2test -v(two tests), and/privatetemporary-directory mount normalization. No unrelated portability fixes included.