Research: semantic subgroup analysis for Own.NET diagnostic quality (bounded pilot) #427

Classification

• Type: bounded research / evaluation-infrastructure spike
• Priority: P2 / backlog; not a new diagnostic or production analyzer change
• Area: Own.NET regression/oracle corpus → diagnostic quality reporting
• Inspiration: Trusted-AI/AIF360 (subgroup/intersectional evaluation methodology, not an AIF360 integration)
• Related: Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (semantic mutation testing), Prove value: oracle sweep over new OSS repos + draft triage table #201 (real-OSS oracle), Audit foundation: finding identity v2 (pattern, occurrence, lineage) with provenance #266 (finding identity)

Problem

An aggregate pass rate or precision/recall number can hide systematic failures concentrated in particular C# semantic shapes. For example, a detector may perform well on direct Dispose() but miss guarded release, aliasing, partial-class teardown, or interprocedural ownership. Today we need a reproducible way to ask which semantic subgroups are actually covered, and where verified false negatives/false positives cluster.

This is an evaluation issue, not a claim that statistical subgroup metrics establish analyzer soundness.

Proposed bounded pilot

1. Inventory existing independently reviewed oracle cases (including clean/negative cases); reuse Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 valid mutants if/when available. Do not count an unreviewed synthetic mutant as ground truth merely because its generator intended to inject a defect.
2. Add minimal case-level metadata with stable ID, source/revision, expected behavior, diagnostic family, and a small multi-label semantic taxonomy. Suggested initial tags: partial-type, aliasing, guarded-release, exception-path, async-lifetime, interprocedural. Tags describe observed case semantics, not assumed analyzer capabilities. Start with a handful of populated tags; avoid an exhaustive taxonomy.
3. Run the existing analyzer at a pinned revision, matching diagnostics to expected family and relevant site, not just exit code or warning count. Preserve actual source examples and evidence for every discrepancy.
4. Emit a deterministic JSON/CSV report and a short human-readable table: evaluated cases; TP/FN for known-positive cases; FP/TN for independently reviewed known-negative cases; unknown/unreviewed/invalid/infra cases reported separately. Provide per-tag counts and only a few preselected tag intersections where the denominators support interpretation.
5. Investigate the worst count-backed subgroup(s) manually. Reproduce any new gap and distinguish an existing limitation, duplicate, oracle error, unsupported shape, or genuine previously untracked analyzer miss.

Important denominator rules: one case should not silently turn into multiple observations because multiple diagnostics were emitted; record both case-level outcome and finding-level details. Report raw numerators/denominators for every rate. Do not calculate precision without verified positives and negatives for the relevant population; do not label unreported real-world bugs as true negatives. For tiny slices, display counts and mark rates as insufficient evidence rather than ranking by percentages. Distinguish extraction failures/timeouts from misses.

Scope boundaries

• Standalone/offline evaluator layered on existing fixtures and analyzer entry points; no AIF360 Python dependency.
• No default CI gate, broad instrumentation of OwnIR, automatic issue spam, fairness/protected-attribute analysis, or claim of globally representative recall.
• Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 generates/validates semantic mutants; this issue stratifies oracle-backed results across semantic features. Neither requires the other for the first pilot.
• If there is no defensible mixed positive/negative oracle, explicitly produce a coverage/gap inventory instead of bogus precision/recall.

Deliverables / acceptance

• A small, reviewed baseline corpus with stable case IDs, provenance, expected outcome, and semantic tags (reusing existing evidence where possible).
• Reproducible one-command local run with pinned analyzer revision/toolchain; zero persistent changes to test source.
• Machine-readable outcomes and concise per-tag/per-selected-intersection tables, with explicit denominators and unknown/invalid/infra buckets.
• At least one manually audited apparent weak slice; link reproducer and classify each observed miss honestly.
• Short GO/STOP decision: did subgroup analysis reveal a new actionable blind spot or meaningful regression signal beyond current tests/Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405, at acceptable maintenance cost?

Stop / promotion gate

Timebox as a small spike. Stop if metadata oracles are unreliable, slices are too sparse to interpret, or every identified gap is already known. If useful, follow with narrowly targeted fixtures / mutation operators or an optional regression dashboard; defer CI thresholds until there is a stable, sufficiently populated benchmark.

Success is finding a concrete, validated weakness or demonstrating an efficient reproducible way to prevent regressions, not producing an attractive aggregate score.