Skip to content

feat(v4): PR 3c of the 4.0 cut - remove the ImputationDiD/TwoStageDiD/ContinuousDiD fit-time keywords and relay the bootstrapped surfaces - #865

Merged
igerber merged 2 commits into
mainfrom
feat/v4-pr3c-aggregate-relay
Oct 21, 2026
Merged

igerber merged 2 commits into
mainfrom
feat/v4-pr3c-aggregate-relay

Conversation

@igerber

@igerber igerber commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • 4.0 cut, PR 3c of three (3a test(v4): PR 3a of the 4.0 cut - migrate the suites to the post-fit aggregation containers #863 migrated the suites; 3b feat(v4): PR 3b of the 4.0 cut - remove the fit-time aggregate=/balance_e= keywords on CS, EfficientDiD, StackedDiD, dCDH and HAD #864 removed the CallawaySantAnna / EfficientDiD / StackedDiD / dCDH / HAD keywords). This PR removes the last fit-time aggregate= / balance_e= keywords - ImputationDiD (M-021/M-118), TwoStageDiD (M-022/M-119), ContinuousDiD (M-025) - and flips those five ledger rows to removed; passing the keywords is now Python's native TypeError, and survey_design moves to positional slot 7 on all three. With them go ContinuousDiD's "eventstudy" spelling and fit-time value validation, its dead analytic fit-time inference arm, and ImputationDiD's unreachable pretrends+replicate fit-time gate (M-127 records the exact predicate). TripleDifference's staggered-mode keyword is the one surviving fit-time aggregate= (M-140/M-141).
  • The bootstrapped-fit RELAY (spec section 6 amendment): the deprecated keyword was the only route to bootstrap event-study/group inference on these three estimators, so a fit with n_bootstrap > 0 now ALWAYS materializes the percentile event-study (+ group on Imputation/TwoStage) surfaces at fit - FAIL-SOFT with a one-shot overall-only retry - stores them in the legacy fields, and aggregate('event_study'|'group') relays them with kit-snapshot provenance (NaN df column, vcov None). balance_e on such a fit fails closed; a bootstrapped fit with no stored surface (pre-4.0 pickle, fail-soft drop, ContinuousDiD with no post-treatment cells) raises the shared absent-surface ValueError. Analytic fits are unchanged (legacy fields None, kit recompute). Storage rule, fail-soft retry and the ContinuousDiD no-analytical-fallback clause are stated in the spec, the changelog and the REGISTRY relay Notes.
  • Runtime guidance (practitioner_next_steps, DiagnosticReport, results_base hints, _reporting_helpers, plot_event_study), the llms*.txt guides, REGISTRY (Imputation / TwoStage / Continuous notes + three bootstrapped-fit relay Notes, CS/dCDH relay clauses), REPORTING, variance-conventions, the migration guide (slot shifts, bootstrapped-fits box, Fix cells), the ContinuousDiD API page and tutorial 14 markdown, the performance docs and the two TODO rows describe the relay. docs/v4-design.md section 6 carries the amendment and the HYBRIDS paragraph; section 9 marks sub-item 3c shipped; M-093 gains the producer table of which classes populate the legacy fields.
  • tests/test_aggregate_contract.py: the three remaining Test*Shim classes, fit-time fixtures and fail-closed bootstrap pins are retired for Test*Removal pins (TypeError, signature, positional-slot binding), relay pins (the relayed surfaces equal the stored dicts row-for-row and match the per-backend bootstrap oracles) and failure-injection pins for the fail-soft materialization and the storage rule. The estimator suites, DiagnosticReport, event_study_surface (bootstrapped Imputation / TwoStage / Continuous enrolled as native producers), practitioner, guides and naming-guard tests take their relay pins; the module-level FutureWarning filters for the fit-time route are gone. The committed oracle fixture is unchanged.

Methodology references (required if estimator / math changes)

  • Method name(s): Borusyak-Jaravel-Spiess (2024) imputation estimator; Gardner (2022) two-stage DiD; Callaway-Goodman-Bacon-Sant'Anna continuous-treatment DiD. No estimator mathematics changes: the relay stores and serves the surfaces the pre-existing fit-time bootstrap code computed (pinned against the committed per-backend oracles captured on the pre-removal tree); the analytic recompute paths are untouched.
  • Paper / source link(s): docs/methodology/REGISTRY.md ImputationDiD / TwoStageDiD / ContinuousDiD sections (bootstrapped-fit relay Notes added)
  • Any intentional deviations from the source (and why): None

Validation

  • Tests added/updated: tests/test_aggregate_contract.py (459 tests, numpy and rust backends), tests/test_imputation.py, tests/test_two_stage.py, tests/test_continuous_did.py, tests/test_methodology_imputation.py, tests/test_pretrends_event_study.py, tests/test_survey_phase8.py, tests/test_diagnostic_report.py, tests/test_event_study_surface.py, tests/test_practitioner.py, tests/test_guides.py, tests/test_naming_guard.py. The touched and consumer suites (23 files, 2,856 tests) pass; test_v4_matrix sees the five rows flip (due-at-4.0 non-terminal rows 16 -> 11); the contract file and the three estimator suites also pass under the Python 3.9 CI leg's scipy 1.13.1 / numpy 2.0.2 pins. mypy diff_diff clean; Sphinx -W build clean; tutorial 14 executes under nbmake.
  • Backtest / simulation / notebook evidence (if applicable): N/A

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

Changelog

  • changelog.d/ fragment added (or N/A - no user-visible change): Yes - changelog.d/20261021-v4-pr3c-aggregate-relay.md (### Removed, ### Breaking Changes, ### Changed, ### Internal)

Follow-ups

  • Cut PR 4 (the three merges + tutorials) per docs/v4-design.md section 9.

…/ContinuousDiD fit-time keywords and relay the bootstrapped surfaces

Spec section 9 item 3, sub-item 3c (3a #863 migrated the suites, 3b #864
removed the CS/EfficientDiD/StackedDiD/dCDH/HAD keywords). Ledger rows M-021,
M-022, M-025, M-118 and M-119 flip to removed; M-093 and M-127 are amended.

- diff_diff: ImputationDiD / TwoStageDiD / ContinuousDiD lose the fit-time
  aggregate= (and balance_e=) parameters (ContinuousDiD's "eventstudy" spelling
  and value validation, its dead analytic fit-time inference arm and
  ImputationDiD's unreachable pretrends+replicate fit-time gate go with them).
  A fit with n_bootstrap > 0 ALWAYS materializes the percentile event-study
  (+ group on Imputation/TwoStage) surfaces at fit - fail-soft, with a
  one-shot overall-only retry - and aggregate('event_study'|'group') RELAYS
  them; balance_e on such a fit fails closed; a bootstrapped fit with no stored
  surface raises the shared absent-surface ValueError
  (aggregation.absent_bootstrap_surface_message). Analytic fits are unchanged.
  practitioner / diagnostic_report / results_base hints / _reporting_helpers /
  plot_event_study messages and the llms*.txt guides describe the relay.
- docs: ledger rows + M-093 producer table + M-127 exact predicate, spec
  section 6 amendment + HYBRIDS paragraph + "(SHIPPED, PR 3c)", REGISTRY
  Imputation/TwoStage/Continuous notes + three bootstrapped-fit relay Notes +
  CS/dCDH relay clauses, REPORTING, variance-conventions, migration guide
  (slot shifts, bootstrapped-fits box, Fix cells), ContinuousDiD API page and
  tutorial 14 markdown, performance docs, TODO rows.
- tests: test_aggregate_contract retires the three remaining Test*Shim
  classes, fit-time fixtures and fail-closed pins for Test*Removal pins,
  relay-equals-stored-surface + oracle pins and failure-injection pins for
  the fail-soft materialization; the estimator suites, DiagnosticReport,
  event_study_surface (bootstrapped fits as native producers), practitioner,
  guides and naming-guard tests take their relay pins; the module-level
  FutureWarning filters for the fit-time route are gone.
@github-actions

Copy link
Copy Markdown

Overall assessment: ✅ Looks good

No unmitigated P0 or P1 findings identified in the supplied diff.

Executive summary

  • The affected methods are ImputationDiD, TwoStageDiD, and ContinuousDiD.
  • The bootstrap relays preserve stored inference; no estimator mathematics changes were identified.
  • Missing surfaces, balanced-window requests, and analytical fallback paths follow the documented registry policy.
  • A changelog fragment and substantial regression coverage accompany the behavior changes.
  • Validation was limited to source inspection and syntax checks: numerical dependencies are unavailable, and DNS prevented direct paper retrieval.

Methodology

P3 — Informational: documented relay limitations. Cross-checked the changed paths against the registry, source-linked docstrings, and repository paper analyses covering BJS Theorems 2–3/Test 1, Gardner §3.3, and CGBS binarized aggregation.

Bootstrap surfaces carry no joint covariance and clear per-row df; Imputation’s pretrend leads retain their analytical inference. Bootstrapped balanced-window requests fail closed. These conventions are explicitly documented at docs/methodology/REGISTRY.md:L1665, L2230, and L2325.

  • Impact: Known inference and aggregation limits remain explicit.
  • Concrete fix: None required.

No undocumented weighting, comparison-group, identification, or variance change was identified.

Code Quality

No findings. The new branches handle absent surfaces explicitly and clear incomplete surfaces before analytical fallback. No new unsafe inline inference calculation was identified.

Performance

P3 — Informational: additional bootstrap fit work. Event-study/group materialization adds computation, intentionally documented and quantified in docs/v4-design.md:L635-L677.

  • Impact: Bootstrapped plain fits do more work; analytical defaults retain lazy aggregation.
  • Concrete fix: None required.

Maintainability

No findings. Shared absent-surface guidance is centralized, and the three removed signatures contain no remaining references to the deleted parameters.

Tech Debt

P3 — Informational: replay work is tracked. Balanced-window bootstrap replay and ContinuousDiD post-fit replay remain recorded in TODO.md:L40-L41.

  • Impact: These capabilities remain unavailable through explicit errors.
  • Concrete fix: None required for this PR.

Security

No findings in the changed code.

Documentation/Tests

No actionable findings. The release-note fragment follows changelog.d/README.md. Added tests cover oracle parity, missing surfaces, retry failures, storage rules, positional binding, and downstream consumers.

Syntax checks passed for all 31 changed Python files. Runtime tests could not run because pytest, NumPy, pandas, and SciPy are absent. Primary-paper verification was limited to the repository’s source-mapped reviews because external retrieval failed.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Oct 21, 2026
…bounded columns

The Linux and Windows Rust-backend legs failed one contract pin,
TestTwoStageAggregate::test_bootstrap_prop5_relay_matches_oracle_and_aggregator_warnings_fire:
ts/boot/p5/es[1].t_stat drifted 1.56-1.59e-13 relative from the macOS capture
while att and se each passed at rtol 1e-13. A t-statistic is att/se, so its
relative drift compounds to 2 * rtol to first order; the bootstrapped Prop-5 arm
had never been compared on a non-Accelerate platform before this PR (main pins
only its analytic arm). Apply the derived bound to every oracle t_stat comparison
(event-study, group, overall, total, ContinuousDiD overall) via _t_rtol(); att,
se, p_value and CI tolerances are unchanged.
@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 42803fe34c8b244f29ed07e964695b72e98dada9


Overall assessment: ✅ Looks good

No unmitigated P0 or P1 findings identified. The prior review’s assessment remains unchanged.

Executive summary

  • Affected methods: BJS ImputationDiD, Gardner TwoStageDiD, and CGBS ContinuousDiD.
  • The changed routing preserves the existing bootstrap calculations; no undocumented change to weighting, identification, or variance was identified.
  • Missing surfaces, failed materialization, and balanced-window requests follow the documented registry policy.
  • The latest t-statistic tolerance adjustment keeps separate effect and SE checks intact.
  • Syntax checks passed for all 31 changed Python files, and the changelog convention check passed. Numerical tests could not run in this environment.

Methodology

P3 — Informational: documented bootstrap relay limitations.

Cross-checked the changed paths against the registry, source-linked docstrings, and repository paper analyses covering BJS Theorems 2–3/Test 1, Gardner §3.3, and CGBS binarized event-study aggregation.

The relays preserve stored inference, clear per-row df, and expose no joint covariance. Imputation’s pretrend leads retain analytical inference. Bootstrap balancing restrictions and ContinuousDiD’s absent-surface behavior are explicitly documented at docs/methodology/REGISTRY.md:L1665, docs/methodology/REGISTRY.md:L2230, and docs/methodology/REGISTRY.md:L2325.

  • Impact: These documented limits remain explicit and fail closed where necessary.
  • Concrete fix: None required.

Direct primary-paper retrieval was blocked by DNS; paper verification relied on the repository’s source-mapped analyses.

Code Quality

No actionable findings. The new failure branches clear incomplete surfaces, preserve the documented overall-only retry behavior, and reject unavailable surfaces explicitly. No new unsafe inline inference calculation was identified.

Performance

P3 — Informational: additional bootstrap fit work.

  • Impact: Bootstrapped fits now materialize additional aggregation targets. This cost is documented in docs/v4-design.md:L635-L677; analytical defaults retain lazy aggregation.
  • Concrete fix: None required.

Maintainability

No findings. Absent-surface guidance is centralized, and static inspection found no stale references to removed parameters inside the three changed fit() methods.

Tech Debt

P3 — Informational: replay extensions are tracked.

  • Impact: Balanced bootstrap replay and ContinuousDiD post-fit replay remain unavailable through explicit errors, with follow-ups recorded in TODO.md:L40-L41.
  • Concrete fix: None required for this PR.

Security

No findings in the supplied changes.

Documentation/Tests

No actionable findings. The changelog fragment follows the project convention. Added coverage addresses oracle parity, retry failures, storage rules, positional binding, and downstream consumers.

The t-statistic tolerance change at tests/test_aggregate_contract.py:L79-L90 accommodates compounded ratio rounding while retaining independent effect/SE checks.

Validation completed: syntax checks for 31 changed Python files and changelog_compile.py check. Runtime tests were unavailable because pytest, NumPy, pandas, and SciPy are not installed.

@igerber
igerber merged commit 066da1a into main Oct 21, 2026
37 checks passed
@igerber
igerber deleted the feat/v4-pr3c-aggregate-relay branch October 21, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant