Research: BugBug-inspired test-impact selection and shadow scheduler evaluation (bounded spike) #437

Classification

• Type: bounded research / test-infrastructure evaluation spike
• Priority: P2 / backlog; no production CI test skipping by default
• Area: Own.NET changes → impacted diagnostics/components → regression/mutation/oracle cases → test execution and evidence
• Prior art: mozilla/bugbug (especially testselect, regressor, testfailure); Mozilla's 2020 test-selection write-up
• Related, not replaced: Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (semantic mutation generation and analyzer detection), Research: semantic subgroup analysis for Own.NET diagnostic quality (bounded pilot) #427 (oracle-backed semantic subgroup evaluation), Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432 (refactoring-based metamorphic controls), Research: fault-injection tests for CLI failure visibility and SARIF artifact integrity (bounded pilot) #428 (CLI fault injection), P-022 diagnostics: normalized AnalysisTrace and first-divergence minimizer for Python↔Rust shadow mode #269 (analysis-trace divergence), Prove value: oracle sweep over new OSS repos + draft triage table #201 (real-OSS independent oracle)
• Default decision: borrow the evaluation methodology, not BugBug's Mozilla-specific pipeline, ML models, infrastructure, or CI integration.

Problem

We can run an analyzer's test suite and see it go green without learning which ownership rules/IR changes are insufficiently tested. Conversely, exhaustively running every regression, semantic mutant, real-OSS case, and future expensive oracle for every small change could become unnecessarily costly.

The useful BugBug idea here is change-aware test selection with measured regression-detection cost, not automatically introducing XGBoost. Mozilla links historical patch/test outcomes, evaluates predictions with a time-respecting split, and uses shadow schedulers before changing actual test scheduling. Its 2020 report of large task savings is a historical Firefox-scale outcome, not an expected benefit for Own.NET.

Research question: Can a cheap, deterministic Own.NET-specific test-impact/prioritization heuristic provide useful earlier feedback or lower optional test execution cost without missing independently verified analyzer regressions, compared with the existing full suite and simple baselines?

Distinct scope versus existing issues

• Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 owns mutation operators, validation of valid/equivalent/invalid mutants, independent ownership-defect oracle, and analyzer detection classification. This issue consumes those verified case outcomes if available; it must not add a second mutation harness or claim mutation kills from compiler failures.
• Research: semantic subgroup analysis for Own.NET diagnostic quality (bounded pilot) #427 owns diagnostic-quality subgroup reporting. This issue measures which tests should run for a given change and what regressions a proposed test set would miss.
• Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432 owns metamorphic/refactoring test generation and correspondence. Reuse its independently reviewed cases where helpful; do not build an AST-diff/refactoring engine.
• Do not confuse the two different meanings of mutation adequacy: (a) whether Own.NET detects injected ownership defects (Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405), and (b) whether a test/scheduler selection runs the test that would expose an analyzer regression (this issue). They require separate denominators and evidence.

Bounded pilot

0. Inventory and baseline

At a pinned Own.NET revision, map the current test entry points and rough wall-clock costs. Inventory available links between:

• changed source modules / Roslyn extractor facts / OwnIR lowering / MOS or abstract-state logic / diagnostic family;
• stable test IDs and fixture provenance;
• known regressions, soundness-sensitive cases, valid mutants from Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (if any), and negative controls;
• CI job/configuration differences if materially relevant.

Do not infer mappings just from matching file names when a semantic dependency can cross modules or an interprocedural boundary. Keep an explicit conservative fallback for shared infrastructure changes.

1. Compare simple selectors first

Implement, or prototype offline, three strategies on the same change/test evidence:

1. Full baseline: current relevant complete suite, never silently filtered.
2. Simple baseline: changed-directory or manually maintained rule→test mapping, with explicit required smoke/soundness sentinel cases.
3. Conservative impact heuristic: union of known ownership diagnostic/rule dependencies, extractor/OwnIR/MOS dependencies, historical co-failure/co-change data only if trustworthy, and a mandatory fallback for unmapped changes.

Each candidate returns a ranked/selected set with reasons and an unresolved/unmapped bucket. Do not present source-code distance as proof of semantic irrelevance.

2. Shadow evaluation only

For a small, reviewed sample of real past changes with independently confirmed failing tests/regressions, and/or purpose-built verified regression replays:

• replay in temporal order; only use metadata available before the evaluated change;
• compute the selected tests without actually suppressing tests;
• compare against the complete tests that did run or against purposefully backfilled full-test results;
• explicitly mark unknown outcomes when a test was not run. Never turn not run into pass;
• distinguish a patch introducing an analyzer false negative, false positive, diagnostic-location regression, extraction failure, infrastructure failure, and flaky/unstable test;
• record per-case missed regressions and time/cost delta; show counterexamples, not just aggregate percentages.

If no credible historical regression labels exist, do an offline fixture/replay exercise and report that its detection-rate estimate is not predictive of future commits. A synthetic corpus is acceptable for checking wiring, not for claiming real-world scheduler effectiveness.

3. Optional, gated follow-up only

Consider a statistical model (BugBug-like patch/test pairs), learned risk score, additional test prioritization, or CI adoption only after measurable dataset sufficiency, leakage-safe evaluation, and incremental benefit over the deterministic baseline have been demonstrated.

Mozilla's Hackbot reviewer test-validation proposal #7128 is interesting related work (reproduce a bug, test adequacy via mutants, independently derive negative examples), but is an upstream proposal rather than shipped evidence. Any independent test-adequacy oracle belongs in #405/#432 or a separately justified follow-up, not in this scheduler pilot.

Required artifacts and metrics

• Reproducible single local invocation; pinned commit/toolchain/test-config hashes; unchanged working tree after replay.
• Machine-readable JSON/CSV plus a short readable report with change_id, test_id, selector reason, executed/not-executed, observed outcome, independent oracle provenance, elapsed time, and status (verified, unknown, invalid, flaky, infra-error).
• Detection: numerator/denominator of independently verified regressions surfaced by the proposed selection, with every miss linked to a reproducer; unknown denominators explicitly excluded.
• Cost: test-count and measured runtime savings versus full baseline (with methodology; no extrapolating from test counts alone).
• Latency: time-to-first-actionable failure when order/priority is evaluated.
• Risk: coverage of must-run soundness sentinels and unmapped or shared-pipeline changes; no suppression of full-run failures in shadow mode.
• Per-class summaries only where sample sizes allow; do not treat an aggregate score as a soundness argument.

Deliverables / acceptance

• Current test layout and actual baseline costs measured on a pinned revision.
• At least two simple, auditable selectors compared against full-suite baseline; rules, fallbacks, and reasons documented.
• A small historical/replay set with explicit provenance and outcome availability; no leakage from future failures or not run cases.
• Shadow report listing every verified miss, not merely a headline recall rate, with its source/fixture and the reason it was omitted.
• Verified separation of analyzer-detection (semantic mutants) from test-selection detection; no fabricated success from compiler failures, unrelated diagnostics, or missing outcomes.
• GO/STOP conclusion with measured incremental value, runtime savings, newly exposed test-mapping weaknesses, failure modes, and maintenance cost.
• No default CI test suppression, no unreviewed ML model, no new soundness claim.

Stop / promotion gate

STOP if the test suite is currently too cheap to warrant scheduling, the history lacks reliable full-test outcomes, selector results are indistinguishable from a simple directory/rule mapping, or any observed miss is too difficult to explain/contain. Report the negative result rather than introducing infrastructure.

A worthwhile GO for a narrowly scoped follow-up requires reproducible savings or earlier actionable failures with no verified regression misses on the pilot corpus, conservative behavior for unmapped changes, and a clear plan for continued full-suite runs/backfill. Zero observed misses in a small pilot is not a safety guarantee.

Out of scope: wholesale BugBug/Hackbot port, Bugzilla integration, permanent CI skipping, auto-merge or auto-repair agents, new diagnostic rules, mandatory XGBoost/OR-Tools, or claiming Own.NET soundness from a trained classifier.