Research: fault-injection tests for CLI failure visibility and SARIF artifact integrity (bounded pilot) #428

Classification

• Type: bounded research / resilience-test infrastructure spike
• Priority: P2 / backlog; do not implement a production fault-injection framework by default
• Area: public owen / own-check launch path → Roslyn facts handoff → Rust CLI → rendered output (especially SARIF, where supported)
• Inspiration: liblit/Murphy (syscall-level fault injection, deterministic replay and reduction); modern Linux strace --inject as one optional experimental mechanism, not a required dependency
• Related: P-022 step 7b: the production Rust OwnIR executable (own-cli ownir) — command, output and exit-code parity behind the existing launcher #261 (existing panic/uncatchable-death + CLI exit contracts), P-022 step 8: Rust-default cutover, rollback gate, and Python distribution removal #262 (public launcher and unexpected child-status contract), Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (semantic mutation testing; distinct problem), Экстрактор падает на методе с 31+ условиями if/else if при --flow-locals (JSON MaxDepth 64), весь репозиторий остаётся без фактов #401 (real-world whole-run extractor failure as motivation, not a new reproduction claim here)

Problem

A static analyzer can be semantically correct on valid inputs yet produce a dangerously misleading outcome when its surrounding pipeline fails:

• facts are absent, truncated, unreadable or partially written;
• a child process terminates unexpectedly;
• output serialization/write fails, or a previous report survives a failed run;
• the launcher loses the failure status and presents a clean run / empty SARIF as if the analysis completed.

The safety property is failure visibility and artifact integrity, not ownership-analysis soundness. 0 findings after successful analysis and 0 findings because the analysis never completed must not be indistinguishable.

#261 already tested Rust CLI catchable panic and hard-death controls behind an opt-in fault-injection feature. Do not redo those tests. This issue probes external I/O failures and end-to-end output handling, especially at the extractor → facts → analyzer → report boundaries.

Scope: small read-only-to-product pilot

First inventory the actual current production invocation, supported formats and exit/status contracts on Linux/Windows from the repository and tests. Do not assume that every entry point writes SARIF to a file (some may render to stdout), or that an illustrative syscall number maps to the desired artifact.

Select 4–6 reproducible, independently controllable scenarios from:

1. Input read/open failure: valid input becomes inaccessible or unreadable after preflight; check stage-specific error classification.
2. Facts handoff failure: simulated missing/truncated/unreadable facts after extraction; verify no downstream clean result, and no silent fallback.
3. Report destination failure: cannot create/open the chosen output path (permissions or deliberately invalid path, using the platform's actual supported output mode).
4. Write failure after partial progress: injected ENOSPC/EIO or a controlled failing writer targeted to the report, not an arbitrary fifth write; verify no successfully published truncated report.
5. Interrupted/short output: if realistically inducible on a supported path, verify the result is either retried correctly or reported as failure.
6. Stale artifact hazard: leave a valid report from an earlier run, then fail the next run at the producer; no stale artifact should be presented/consumed as the current result without explicit provenance.

Include two controls: a genuinely clean successful analysis and a successful analysis with real findings, so the harness cannot count every nonzero exit as an expected failure.

For errors already covered by contract tests, reuse those fixtures instead of adding copies.

Mechanism: prefer the narrowest deterministic hook

• First try existing test seams, temporary directories, permission controls and injected stream/writer failures where applicable.
• On Linux, strace --inject can be evaluated for a single experiment. Prove the intercepted syscall, process, file descriptor and artifact being targeted; -f --inject=write:...:when=5 by itself is not sufficient evidence that the SARIF write was faulted.
• Keep Windows as a contract-level validation target using portable test seams. Do not turn Linux ptrace into a required Windows/CI dependency.
• Preserve production semantics; any new fault hook must be test-only, opt-in and disabled in normal builds/runs. No broad interception framework or Murphy vendoring.
• Pin the scenario and any injection plan/seed, and make it rerunnable without network access. Minimize a complex failure to the first causally necessary failing operation if needed, borrowing Murphy's replay/reduction idea.

Oracle and evidence

For each scenario record:

Do not classify a scenario as handled merely because the process exited nonzero. Check that the launcher propagates an actionable failure and downstream consumers cannot misinterpret produced or pre-existing artifacts as successful current results.

Distinguish injection-not-triggered from pass. An unsupported injection must not be scored as coverage. Do not bake in a universal numeric exit code; follow existing platform/entry-point contracts (#261/#262) unless a separately approved change is necessary.

Deliverables / acceptance

• Current end-to-end error/output contract mapped and existing relevant tests (P-022 step 7b: the production Rust OwnIR executable (own-cli ownir) — command, output and exit-code parity behind the existing launcher #261/P-022 step 8: Rust-default cutover, rollback gate, and Python distribution removal #262) identified, without duplicating them.
• 4–6 small deterministic fault scenarios + 2 success controls, selected based on actual reachable code paths.
• One local command that runs the pilot and emits human-readable + machine-readable evidence, including injection-fired checks and artifact hashes.
• For each case, evidence demonstrates which operation was faulted and whether any stale, truncated or falsely clean report could be observed.
• Runs leave no modified source files or misleading persistent output artifacts; any unavoidable platform limitation is documented.
• Short conclusion separates confirmed integrity/failure-propagation bugs, already-covered behavior, inconclusive injections, and unsupported cases; link a minimized reproducer for each new bug.

Stop / promotion gate

P2 research only. Stop after the bounded pilot. Promote to a permanent regression test or open a narrowly scoped fix issue only if a new, reproducible failure-masking or artifact-integrity bug is found, or existing critical failure contracts demonstrably lack coverage. If all useful paths are already protected, preserve the evidence and STOP, rather than developing a reusable fault-injection platform.

Explicit non-goals

• Improving OWN diagnostic recall/precision or proving analyzer soundness.
• Replacing or widening Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 semantic mutation testing.
• Repeating P-022 step 7b: the production Rust OwnIR executable (own-cli ownir) — command, output and exit-code parity behind the existing launcher #261's panic/hard-death qualification.
• A full syscall-fuzzing campaign, production dependency on Murphy/strace, or platform-wide chaos testing.
• Changing public exit codes or output formats without an independently justified contract decision.