Repository navigation
Harden benchmark runners and pipelines - #162
Merged
Jake Bailey (jakebailey) merged 7 commits intoOct 20, 2026
Merged
Conversation
Approved compiler builds can still produce unexpected files. Artifact handling must not follow links into the agent filesystem, block on special files, or expose logging commands through filenames in upload tasks. Apply the same checks to packaged API artifacts and require a regular compiler manifest before identifying the implementation.
Scenario installers and compiler runs share source trees with later host-side setup. Protect Git metadata from installer writes and keep hooks disabled so a later checkout cannot execute retained repository configuration as an unintended host-side action. Pass the selected scenario commit through a validated metadata handoff rather than giving general setup output variable-publication authority.
Downloaded compiler artifacts and unscheduled files must not become additional report inputs. Use the setup matrix as the source of expected result names, including API jobs, and reject missing or nonregular files. Report labels belong in literal arguments, not shell source. Keeping filenames and labels as data avoids broad artifact globs and shell interpretation while preserving the original benchmark text.
Rendering benchmark data does not require blob-write credentials. Keep upload authority in a separate job whose eligibility comes from setup, without treating job separation as authentication of the reported data. Reporting needs a GitHub token only when posting a PR comment. Limit that token to pull-request write permission, create it after result processing, and avoid revocation attempts when token creation was skipped or failed.
Compiler and benchmark text must not gain pipeline authority simply because the agent recognizes a logging command in it. Agent-enforced step capabilities are a better boundary than repeatedly rewriting output. Intentional variable publication needs a narrower boundary than general data processing. Required metadata and result handoffs let later steps detect missing output even if an earlier task reports success prematurely. Preserve original benchmark text while retaining protocol escaping and inert diagnostics where a step is authorized to publish variables.
Benchmark hosts let every user run pyperf as root with any arguments. pyperf can launch arbitrary commands, so any code running as the agent user, including a compiler under test, could become root and persist beyond the job. The pipeline only needs one tuning operation with host-specific CPUs, so grant the agent user exactly that and nothing else.
Maintainer approval remains the primary trust decision. Isolation adds protection if harmful compiler or dependency behavior slips through. Even without sudo, code running directly as the agent user can read agent credentials, reach the network, and modify the agent, cached hosts, or scenario repositories to affect later jobs. Running builds and benchmarks in bubblewrap confines them to the files each step needs and prevents privilege elevation, without the runtime overhead of a VM or gVisor. API packaging and consumer subprocesses need the same boundaries.
Jake Bailey (jakebailey)
deleted the
harden-benchmark-artifact-handling
branch
October 20, 2026 18:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While we only run benchmarks on trusted PRs, do a bit of work to harden the pipeline against AzDO commands, and sandbox the builds/benchmark runs.