Repository navigation
Conversation
Merging this PR will not alter performance
|
1ff3345 to
33db967
Compare
Bump instrument-hooks to protocol version 3. Integrations timestamp their
OpenTelemetry spans and logs with the monotonic clock of the benchmark
markers, shifted by `instrument_hooks_monotonic_offset` to read as wall
clock times. In analysis mode, the hooks now send that offset to the
runner in a new `SetTelemetryMonotonicOffset { pid, offset_ns }` command,
so that the runner can place the telemetry back on the clock of the
markers.
The runner accepts protocol version 3 and acknowledges the command in
every executor. Executors that don't collect telemetry ignore it.
Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the files holding the OpenTelemetry spans and logs the benchmarks emit, shared by the runner that writes them and the parsers that read them: `otel/spans.msgpack.zst` and `otel/logs.msgpack.zst` in the profile folder. Each file is a zstd stream of positional MessagePack records, one per span or log, like the memtrack artifacts. Rows are flattened for the tables they are loaded into: ids as hex, attribute values as text, the benchmark in its own field, and the span or log time relative to the benchmark start when known. zstd runs with a 16 MB window and long-distance matching, since rows repeat their resource attributes far apart. Versions follow `UnwindData`: each record is wrapped in a versioned enum (`SpanRecord`, `LogRecord`), `SpanRow` and `LogRow` alias the current version, and older versions are upgraded through `From`. Version 1 fixtures must keep decoding. `TelemetryReader` decodes a file from any `Read`, such as an archive entry, as its bytes arrive. Unlike a truncated memtrack stream, a file cut in the middle of a row ends with an error. Refs COD-3863 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `codspeed run --mode telemetry`, an experimental mode collecting the OpenTelemetry spans and logs the benchmarks export. It is a separate pass, so nothing has to keep the export out of the measured runs, and the benchmarks run once each instead of once per walltime iteration. The mode runs on a new analysis executor: it runs the benchmarks once each without a profiler, answering the integration as an analysis mode, with analysis instruments hooked onto it. An instrument sets up before the process is spawned, gets the integration's start and stop profiler commands, and stops once the process exits, with the benchmark markers. The executor saves those markers as `ExecutionTimestamps.msgpack` in the profile folder, and fails when no benchmark ran, unless `--allow-empty` is set. Uploads carry `executor: "analysis"` and the enabled instruments, so later analysis instruments only add an instrument. The telemetry mode enables the telemetry instrument: - It starts an OTLP/HTTP receiver on a free localhost port, passed to the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports, plain or gzipped, and writes each trace and log request to disk as received. Metrics are acknowledged and dropped. - Once the process exits, the requests are decoded and written as the runner-shared telemetry files, one row per span or log, formatted like the OpenTelemetry Collector's ClickHouse exporter. A request that fails to decode is skipped with a warning. - Each row is placed on the clock of its benchmark: the clock offset the benchmark process sent in `SetTelemetryMonotonicOffset` is subtracted from its timestamp, and the result is measured from the start marker of the benchmark named by its `codspeed.benchmark.uri` attribute. The offset is left out when either is missing. With several benchmark processes, the first offset is used, with a warning when another differs from it by more than 1 ms. `CODSPEED_RUNNER_MODE` is `telemetry` in this mode, which `cargo codspeed run` reads to pick its telemetry build. exec-harness runs it like memory mode. Refs COD-3863 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Build the divan `otel_example` of a codspeed-rust checkout next to this repository with `cargo codspeed build -m telemetry`, run it through `codspeed run --mode telemetry`, and read back its telemetry files: they hold the benchmark's spans and logs, each tagged with a benchmark that ran and placed within its benchmark's sample. The test is ignored by default since CI has no codspeed-rust checkout. Run it with `cargo test --test otel -- --ignored`. `scripts/run-otel-example.sh` runs the same example and uploads it. Refs COD-3863 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
33db967 to
cf0d278
Compare
Add an experimental
--experimental-otelflag (CODSPEED_EXPERIMENTAL_OTEL) that collects the OpenTelemetry data the benchmarks export, and uploads it with the results.When the flag is set, each run part starts an OTLP/HTTP receiver on a free localhost port and passes its URL to the benchmark in
CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports of traces, logs and metrics, plain or gzipped, on/v1/{traces,logs,metrics}.While the benchmark runs, the receiver only writes each request body to disk as received, so it adds no compression work next to the measured process. A request is stored before it is acknowledged, so everything the process sent before exiting is kept. Once the process exits, the requests are decompressed and packed into a single
otel.tar.zstin the profile folder. Compressing them as one stream with long-distance matching takes advantage of the content that repeats across export batches. Nothing is written when the benchmark exports nothing.To put the spans and logs on the same clock as the benchmark markers, instrument-hooks is bumped for
instrument_hooks_monotonic_offset: integrations timestamp with the monotonic clock shifted to read as wall clock, and record the offset in thecodspeed.clock.monotonic_offset_nsresource attribute. The bindings expose it asInstrumentHooks::monotonic_offsetandMONOTONIC_OFFSET_ATTRIBUTE.tests/otel.rsruns the divanotel_exampleend to end throughcodspeed run --experimental-otelin walltime mode, and checks that every span and log record, shifted back by its resource's offset, falls within the benchmark markers. It needs a codspeed-rust checkout next to this repository, so it is ignored by default:cargo test --test otel -- --ignored. SetCODSPEED_OTEL_TEST_PROFILE_FOLDERto keep the profile folder for inspection.Stacked on #587.
Closes COD-3863