diff --git a/TODO.md b/TODO.md index 6f36089a..1af3e968 100644 --- a/TODO.md +++ b/TODO.md @@ -37,8 +37,8 @@ Related tracking surfaces: | Opt-in singleton-group pruning for TwoWayFixedEffects (static + event-study mode; reghdfe parity): singleton units/periods are currently RETAINED class-wide - the within-demeaned row is zero so points are unchanged, but N/G/residual-df count it and CR1/finite-sample SEs shift (~0.41019 -> 0.40962 measured; REGISTRY "Deviation from R" Note, R5 review) - reghdfe iteratively drops singletons by default while fixest retains them (diff-diff matches fixest); an opt-in knob needs iterative unit+period pruning with consistent cluster/survey/replicate/Conley array subsetting and a default-flip decision protocol (moves published SEs) | `diff_diff/twfe.py`, `diff_diff/estimators.py`, `diff_diff/utils.py` | 3(a) R5 | Mid | Low | | Cohort-timing validation input for the simultaneous-adoption event-study family (TWFE `event_study=True` + MultiPeriodDiD through 3.9): an optional `first_treat=`/`cohort=` column so simultaneous adoption becomes checkable under the contract-valid time-invariant `D_i` indicator - today the staggered-adoption advisory derives timing from within-unit 0->1 transitions, so it can only fire on off-contract time-varying `D_it` input, and with valid `D_i` adoption timing is not observable in the inputs at all (REGISTRY "staggered-adoption detection limit" Notes, both sections); design questions: validate-only vs steering error, and interplay with the M-011 removal | `diff_diff/twfe.py`, `diff_diff/estimators.py` | 3(a) R2 | Mid | Medium | | CS parity with DMLDiD's replicate conventions: per-cell replicate SEs for `CallawaySantAnna` (CS leaves per-cell SEs on the weighted sqrt-sum under replicate designs; DMLDiD now computes them via `compute_replicate_if_variance`) AND flip the replace-style effective-df relays to min-cap (`min(df_survey, n_valid - 1)`) at all five sites: CS overall (`staggered.py` overall relay), the shared event-study/group aggregation sites (`staggered_aggregation.py` `df_survey_val = min(non_none_dfs)` x2), and the staggered-DDD engine twins (`_staggered_triple_diff_engine.py` overall + event-study) — the replace convention can RAISE df above the QR-rank design df (anti-conservative); latent, fires only when `n_valid < R` (all-zero/non-finite replicate column). DMLDiD's min-cap + the pinned inherited ES/group behavior are the templates (REGISTRY DMLDiD replicate Note) | `diff_diff/staggered.py`, `diff_diff/staggered_aggregation.py`, `diff_diff/_staggered_triple_diff_engine.py` | DML replicate PR | Mid | Low | -| ImputationDiD/TwoStageDiD `aggregate()` recompute levels on bootstrapped fits fail closed ('simple' relays since the M-027 per-level convergence; M-021/M-022); ImputationDiD's per-target psi machinery makes seeded replay tractable (the panel-backed kit retains everything the psi precompute reads), TwoStageDiD's per-level GMM scores are function-locals and would need retention | `diff_diff/imputation_results.py`, `diff_diff/two_stage_results.py`, `diff_diff/aggregation.py` | 2(b) PR-3b | Mid | Low | -| ContinuousDiD `aggregate('event_study')` on bootstrapped fits fails closed (M-025); a seeded post-fit bootstrap-ES replay is tractable - the multiplier draws are seeded (`np.random.default_rng(self.seed)`) - but needs the FULL per-cell `_bootstrap_info` (bread/ee_treated/Psi_eval/dPsi_*/beta_pred) the pruned kit deliberately drops, so shipping it means a kit-payload change with its own memory contract | `diff_diff/continuous_did_aggregation.py`, `diff_diff/continuous_did_results.py` | 2(b) PR-3c | Mid | Low | +| `balance_e` replay on bootstrapped ImputationDiD/TwoStageDiD fits: the unrestricted ES/group surface is relayed since 4.0 PR 3c (computed at fit, `balance_e=None`; M-021/M-022), but `aggregate('event_study', balance_e=k)` still raises there - a balanced-window replay needs retained per-target psi (ImputationDiD: the panel-backed kit retains everything the psi precompute reads, so a seeded replay is tractable) / retained per-level GMM scores (TwoStageDiD: function-locals today) | `diff_diff/imputation_results.py`, `diff_diff/two_stage_results.py`, `diff_diff/aggregation.py` | 2(b) PR-3b | Mid | Low | +| ContinuousDiD post-fit bootstrap-ES REPLAY: since 4.0 PR 3c a bootstrapped fit relays the event-study surface computed at fit (M-025), so the gap is a genuine post-fit replay (fresh draws, no fit-side cost, a route for pre-4.0 pickles); a seeded replay is tractable - the multiplier draws are seeded (`np.random.default_rng(self.seed)`) - but needs the FULL per-cell `_bootstrap_info` (bread/ee_treated/Psi_eval/dPsi_*/beta_pred) the pruned kit deliberately drops, so shipping it means a kit-payload change with its own memory contract | `diff_diff/continuous_did_aggregation.py`, `diff_diff/continuous_did_results.py` | 2(b) PR-3c | Mid | Low | | practitioner `step_name="heterogeneity"` producer-side collisions: three OTHER estimators' advice steps reuse the key with non-heterogeneity labels (`:975` ContinuousDiD dose-response, `:1022` Triple placebo-group, `:1413` LPDiD WAS arrays), so DiagnosticReport's heterogeneity completion silently drops that unrelated advice from `next_steps` via `_filter_steps` - the same latent collision fixed for StackedDiD in M-024 (renamed to `sub_experiment_balance`). Renaming these changes those estimators' report output; audit + rename with per-estimator pins. | `diff_diff/practitioner.py` | 2(b) PR-2 review R9 | Quick | Low | | PreTrendsPower `violation='linear'` on CS `base_period='varying'` input targets the wrong alternative: `δ_pre = M · \|t\|` assumes level coefficients against a common reference, but varying-base pre-treatment effects are consecutive-period comparisons (constant increments under a linear trend). Both CS-sourced routes now WARN (REGISTRY PreTrendsPower Note), and universal-base GAPPED grids fail closed via the `reference_event_times` common-reference guard; what remains is the varying-base resolution - either transforming the violation vector through each coefficient's actual base mapping (needs per-horizon base provenance) or requiring `base_period='universal'` for the linear benchmark - a per-estimator methodology decision with a hand-calculated linear-violation gate | `diff_diff/pretrends.py` | 2(b) PR-1 R5 | Mid | Medium | | `run_placebo_test`/`run_all_placebo_tests` expose ONE overloaded `time` param (calendar for timing/group tests, 0/1 post dummy forwarded into the M-137/M-138 callees - works only because a two-period 0/1 calendar column is both); redesign the wrapper signatures (dual `time=`/`post=` params or a split) so each callee gets an honestly-named argument | `diff_diff/diagnostics.py` | 2(c)-ii PR-B | Mid | Medium | diff --git a/benchmarks/python/benchmark_imputation.py b/benchmarks/python/benchmark_imputation.py index 6161e4d6..b71540b4 100644 --- a/benchmarks/python/benchmark_imputation.py +++ b/benchmarks/python/benchmark_imputation.py @@ -81,7 +81,7 @@ def main(): first_treat="first_treat", ) # Post-fit event study stays inside the timed region (the 3.x - # fit(aggregate="event_study") route, deprecated since 3.9 and removed at the 4.0 cut). + # fit(aggregate="event_study") route, removed at 4.0). es = results.aggregate("event_study") estimation_time = estimation_timer.elapsed diff --git a/changelog.d/20261021-v4-pr3c-aggregate-relay.md b/changelog.d/20261021-v4-pr3c-aggregate-relay.md new file mode 100644 index 00000000..e3010870 --- /dev/null +++ b/changelog.d/20261021-v4-pr3c-aggregate-relay.md @@ -0,0 +1,85 @@ +### Removed +- **The last fit-time `aggregate=` / `balance_e=` keywords are gone (ledger rows flipped to + `removed`).** `fit(aggregate=)` is removed from `ImputationDiD` (`M-021`), `TwoStageDiD` + (`M-022`) and `ContinuousDiD` (`M-025`), `fit(balance_e=)` from `ImputationDiD` (`M-118`) + and `TwoStageDiD` (`M-119`). With PR 3b's removals on CallawaySantAnna / EfficientDiD / + StackedDiD / dCDH / HAD this completes the aggregate-postfit group: every adopter aggregates + through post-fit `results.aggregate(type, balance_e=)` only, and `TripleDifference`'s + staggered-mode keywords are the one surviving fit-time `aggregate=` (`M-140`/`M-141`, + canonical until the DDD container unification). With the keywords go ContinuousDiD's + no-underscore `"eventstudy"` spelling and its fit-time value validation, the dead analytic + fit-time event-study inference arm in `continuous_did.py`, and ImputationDiD's unreachable + pretrends+replicate fit-time gate. + +### Breaking Changes +- Passing `aggregate=` or `balance_e=` to `ImputationDiD.fit`, `TwoStageDiD.fit` or + `ContinuousDiD.fit` now raises Python's native `TypeError: ... unexpected keyword argument`. +- **Positional slots after the removed parameters shift left.** `ImputationDiD` and + `TwoStageDiD`: `survey_design` 9 -> 7; `ContinuousDiD`: `survey_design`, `covariates` + 8/9 -> 7/8. Pass these by keyword. +- `aggregate('event_study', balance_e=k)` on a BOOTSTRAPPED `ImputationDiD` / `TwoStageDiD` + fit raises `NotImplementedError` (the relayed surface was computed unrestricted; re-fit with + `n_bootstrap=0` for a balanced window). Analytic fits recompute with `balance_e` as before. +- A bootstrapped `ImputationDiD` / `TwoStageDiD` / `ContinuousDiD` fit WITH bootstrap inference + but WITHOUT a stored surface - a 3.9-3.13 bootstrapped plain-fit pickle, or a fit whose + fail-soft materialization dropped it - raises `ValueError` from `aggregate('event_study')` / + `aggregate('group')` (previously `NotImplementedError` from the bootstrap gate); the message + names the refit. A 3.9-3.13 bootstrapped pickle fitted with `balance_e=k` stores a restricted + surface and the relay returns it as stored. + +### Changed +- **Bootstrapped `ImputationDiD` / `TwoStageDiD` / `ContinuousDiD` fits relay their event-study + (and, on Imputation/TwoStage, group) surface.** The deprecated keyword was the only route to + bootstrap event-study inference there (no per-draw state is retained for a replay), so a fit + with `n_bootstrap > 0` now ALWAYS computes those surfaces at fit - exactly what + `fit(aggregate="all")` computed, `balance_e=None` - stores them in the legacy + `event_study_effects` / `group_effects` fields, and `aggregate('event_study')` / + `aggregate('group')` RELAYS them (kit-snapshot alpha / anticipation / survey metadata; NaN + `df` column; `vcov` `None`) instead of raising `NotImplementedError`. Analytic fits are + unchanged (legacy fields `None`, kit recompute on demand). Storage rule: the legacy fields + hold a surface IFF the fit carries bootstrap inference. On Imputation/TwoStage a bootstrapped + fit that ends analytical (zero valid draws, total bootstrap failure) clears them and + `aggregate()` recomputes analytically. ContinuousDiD's bootstrapped kit is scalars-only, so + no analytical fallback exists there: a bootstrapped fit with no post-treatment cells never + runs its bootstrap (overall ATT already NaN with a `UserWarning`), stores no surface, and + `aggregate('event_study')` raises the absent-surface `ValueError`. + - Materialization is FAIL-SOFT with a one-shot overall-only retry: an exception while + building the surfaces, or a bootstrap attempt that raises / returns `None` with surfaces + present, leaves them `None`, emits one `UserWarning` ("... could not be materialized ...") + and re-runs the bootstrap overall-only (a seeded fit reproduces the first attempt's weight + draws), so a plain bootstrapped fit's headline quintet and its success/abort outcome are + unchanged by the removal. ContinuousDiD bootstrap failures still propagate as before. + - Consequences on bootstrapped plain fits: the aggregators' own `UserWarning`s (ImputationDiD + Prop-5 unidentified horizons, empty horizons, LOO single-unit auxiliaries; TwoStageDiD + Stage-1/Stage-2 singular or rank-deficient fallbacks) and the per-target bootstrap-stats + `RuntimeWarning`s now fire where a plain fit never emitted them; `summary()` prints the + event-study/group tables and `to_dataframe(level="event_study"|"group")` returns frames + instead of raising; `DiagnosticReport` / `BusinessReport` run heterogeneity (Imputation / + TwoStage on the relayed group dict, ContinuousDiD on the event study) and parallel trends + (Imputation / TwoStage) on the relayed surface instead of skipping with + `NotImplementedError`; `plot_event_study(results)` plots them natively. Measured cost (199 + draws, 300 units x 12 periods): Imputation 0.01s -> 0.08s, TwoStage 0.01s -> 0.04s, + Continuous unchanged at 0.05s; all three default to `n_bootstrap=0`. +- `ImputationDiD`'s inert-`df_convention` warning predicate is now exact: the per-lead inference + is reachable iff `pretrends=True` and the design is not replicate-weighted (`M-127`); the + unreachable pretrends+replicate fit-time gate is deleted (the combination is still rejected + post-fit by `aggregate('event_study')`). +- Runtime messages (`plot_event_study`, the results-class "not stored" hints, + `_reporting_helpers`, `DiagnosticReport` docstrings), `practitioner_next_steps` advice for + bootstrapped ImputationDiD / TwoStageDiD / ContinuousDiD fits, the `llms*.txt` guides, the + ContinuousDiD API page and tutorial, the migration guide and the performance docs describe + the relay instead of steering users to the removed fit-time route. + +### Internal +- `tests/test_aggregate_contract.py`: the remaining `Test*Shim` classes, fit-time fixtures and + fail-closed bootstrap pins are retired for `Test*Removal` pins (TypeError, signature, + positional-slot binding) and relay pins - the relayed surfaces equal the stored legacy dicts + row-for-row and match the per-backend bootstrap oracles - plus failure-injection pins for the + fail-soft materialization and the storage rule. The estimator suites, `DiagnosticReport`, + `event_study_surface` (bootstrapped Imputation / TwoStage / Continuous enrolled as genuine + native producers) and practitioner tests take their relay pins; the module-level + FutureWarning filters for the fit-time route are gone. Naming-guard allowlist entries for the + guides' post-fit `aggregate` vocabulary are finalized. Ledger row `M-093` gains the fit-side + table of which producers populate the legacy fields; `M-127` records the exact predicate. + `docs/v4-design.md` section 6 carries the relay amendment and the HYBRIDS paragraph; section 9 + marks sub-item 3c shipped. diff --git a/diff_diff/_deprecation.py b/diff_diff/_deprecation.py index e061121f..2dc700ff 100644 --- a/diff_diff/_deprecation.py +++ b/diff_diff/_deprecation.py @@ -18,12 +18,12 @@ caller-declared default when neither is passed. - Dropped parameter (no successor name): the parameter keeps its position with a sentinel default; the owner calls ``warn_deprecated_kwarg`` when a - value was actually supplied. (The 3.9 fit-time ``aggregate=`` shims on - CallawaySantAnna, EfficientDiD, StackedDiD, ChaisemartinDHaultfoeuille and - HeterogeneousAdoptionDiD were this shape; they were deleted at the 4.0 cut - (PR 3b) and ImputationDiD / TwoStageDiD / ContinuousDiD follow in PR 3c. - ``warn_deprecated_kwarg`` has no library caller any more (dCDH was the - last) and is retained per section 8 rule 11 for the 5.0 window.) + value was actually supplied. (The 3.9 fit-time ``aggregate=`` shims were + this shape; all of them were deleted at the 4.0 cut - CallawaySantAnna, + EfficientDiD, StackedDiD, ChaisemartinDHaultfoeuille and + HeterogeneousAdoptionDiD in PR 3b, ImputationDiD / TwoStageDiD / + ContinuousDiD in PR 3c. ``warn_deprecated_kwarg`` has no library caller + any more and is retained per section 8 rule 11 for the 5.0 window.) - Renamed results field: the dataclass field takes the new name and ``deprecated_field_property`` builds the read-only warning alias under the old name (a plain class attribute, so it is a descriptor but NOT a diff --git a/diff_diff/_reporting_helpers.py b/diff_diff/_reporting_helpers.py index ab55f42d..58b160b5 100644 --- a/diff_diff/_reporting_helpers.py +++ b/diff_diff/_reporting_helpers.py @@ -50,7 +50,7 @@ def describe_target_parameter(results: Any) -> Dict[str, Any]: - ``ImputationDiD`` / ``TwoStageDiD``: horizon / group tables come from post-fit ``results.aggregate('event_study'/'group')`` since - 3.9 (the fit-time ``aggregate`` kwarg is deprecated) and never + 3.9 (the fit-time ``aggregate`` kwarg was removed at 4.0) and never change ``overall_att``. The headline is always the sample-mean overall ATT (per BJS 2024 Step 3 with ``w_it = 1/N_1``); disambiguate via the event-study or group aggregate if you need @@ -250,7 +250,7 @@ def describe_target_parameter(results: Any) -> Dict[str, Any]: "imputed from a unit+time fixed-effects model fitted on " "untreated observations only (BJS 2024 Step 3). Post-fit " "``results.aggregate('event_study'/'group')`` (3.9; the " - "fit-time ``aggregate`` kwarg is deprecated) yields the " + "fit-time ``aggregate`` kwarg was removed at 4.0) yields the " "horizon / group tables and does NOT change ``overall_att`` " "— for the horizon or group estimand, consult the " "event-study or group aggregate directly." diff --git a/diff_diff/aggregation.py b/diff_diff/aggregation.py index 8cc9ab42..ce29cd70 100644 --- a/diff_diff/aggregation.py +++ b/diff_diff/aggregation.py @@ -1,7 +1,7 @@ """Post-fit aggregation surface (4.0 program Phase 2, spec section 6). -``results.aggregate(type=...)`` replaces the fit-time ``fit(aggregate=)`` -argument (ledger rows M-020..M-027): estimate once, aggregate as a post-fit +``results.aggregate(type=...)`` is the one aggregation surface - the fit-time +``aggregate=`` keyword was removed at 4.0 (ledger rows M-020..M-027): estimate once, aggregate as a post-fit step - the ecosystem's strongest norm (``did::aggte``, ``etwfe::emfx``, Stata ``estat aggregation``). @@ -72,6 +72,24 @@ ) +def absent_bootstrap_surface_message(cls_name: str, level: str) -> str: + """Shared ValueError text for a bootstrapped fit storing no relay surface. + + Bootstrapped ImputationDiD / TwoStageDiD / ContinuousDiD fits materialize + their event-study (and group) surfaces at fit and ``aggregate()`` RELAYS + them (4.0 PR 3c). A fit WITH bootstrap inference but WITHOUT the stored + surface is either a pre-4.0 pickle (a bootstrapped plain fit built none) + or a fit whose fail-soft materialization dropped it; both raise this. + """ + return ( + f"This bootstrapped {cls_name} carries no stored {level} surface: either it " + "was fitted before 4.0 (a bootstrapped plain fit built none) or the surface " + "could not be materialized at fit (see the fit-time warning). Re-fit with " + "diff-diff >= 4.0 for the relayed surface, or with n_bootstrap=0 for the " + "analytical post-fit recompute." + ) + + def resolve_inference_df(results: Any) -> Optional[float]: """Return the degrees of freedom that governed ``results``' overall statistic. diff --git a/diff_diff/continuous_did.py b/diff_diff/continuous_did.py index a4e6e310..48bc263a 100644 --- a/diff_diff/continuous_did.py +++ b/diff_diff/continuous_did.py @@ -17,7 +17,6 @@ import pandas as pd from diff_diff._base import BaseEstimator -from diff_diff._deprecation import NOT_SUPPLIED from diff_diff._dr_scores import drdid_panel_inf_func from diff_diff.aggregation import AggregationKit from diff_diff.bootstrap_utils import ( @@ -101,11 +100,11 @@ def _build_continuous_aggregation_kit( never read the kit at all. On bootstrap fits (``n_bootstrap > 0``) the kit is SCALARS-ONLY: - ``aggregate('event_study')`` fails closed before reading any payload - and the views never read the kit, so a populated payload there would - be pure dead retention. The scalars still distinguish the - bootstrap-NotImplementedError gate from the legacy-pickle no-kit - ValueError. + ``aggregate('event_study')`` relays the fit-time surface (the + percentile event study materialized by ``fit()``) without reading any + payload, and the views never read the kit, so a populated payload + there would be pure dead retention. The scalars still distinguish the + bootstrap relay gate from the legacy-pickle no-kit ValueError. """ is_bootstrap = estimator.n_bootstrap > 0 gt_summary: Dict[Tuple, Dict[str, Any]] = {} @@ -351,7 +350,6 @@ def fit( time: str, first_treat: str, dose: str, - aggregate: Any = NOT_SUPPLIED, survey_design: Optional["SurveyDesign"] = None, covariates: Optional[List[str]] = None, ) -> ContinuousDiDResults: @@ -372,17 +370,6 @@ def fit( First treatment period column (0 or inf for never-treated). dose : str Continuous dose column. - aggregate : str, optional - DEPRECATED (row M-025, removed in 4.0) - aggregate as a - post-fit step instead: ``results.aggregate('event_study')`` - for the binarized event study (underscored - the - ``"eventstudy"`` spelling dies with this parameter), or - ``results.aggregate('dose')`` / ``results.aggregate('simple')`` - views. The dose-response curves and overall ATT/ACRT are - always computed by ``fit()``, so ``aggregate="dose"`` was - already a no-op. Supplying ANY value (including ``None``) - warns ``FutureWarning``; supplied values still run the legacy - routing unchanged until 4.0. survey_design : SurveyDesign, optional Survey design specification for design-based inference. Supports weighted estimation and Taylor series linearization @@ -402,45 +389,13 @@ def fit( ------- ContinuousDiDResults """ - # M-025 deprecation shim: a plain fit() never warns; supplying - # aggregate= with ANY value (None included) warns once, then the - # legacy routing below runs unchanged - "eventstudy" still - # computes the fit-time surface and invalid strings still reach - # the pre-existing ValueError. Only the SENTINEL normalizes to - # None (it would otherwise fail the _VALID_AGGREGATES check on - # every plain fit). The post-fit successor validates its own - # (unified) vocabulary. - if aggregate is not NOT_SUPPLIED: - warnings.warn( - "ContinuousDiD.fit(aggregate=) is deprecated and will be " - "removed in 4.0. Fit once, then aggregate as a post-fit " - "step: results = ContinuousDiD().fit(...); " - "results.aggregate('event_study') (note the underscore - " - "the 'eventstudy' spelling dies with this parameter) / " - ".aggregate('dose') / .aggregate('simple'). The " - "dose-response curves and overall ATT/ACRT are always " - "computed by fit(), so aggregate='dose' was already " - "redundant.", - FutureWarning, - stacklevel=2, - ) - else: - aggregate = None - # Fit-time re-check: __init__ and set_params validate eagerly, so # this only catches DIRECT attribute mutation (est.anticipation = ...) # — an out-of-domain value silently changes the ESTIMAND. The - # assignment also re-normalizes a mutated numpy scalar to int. Placed - # AFTER the deprecation shim so a caller who both mutated and passed - # a deprecated argument still sees the FutureWarning before the raise. + # assignment also re-normalizes a mutated numpy scalar to int. self.anticipation = validate_anticipation(self.anticipation) # 1. Validate & prepare - _VALID_AGGREGATES = (None, "dose", "eventstudy") - if aggregate not in _VALID_AGGREGATES: - raise ValueError( - f"Invalid aggregate: '{aggregate}'. " f"Must be one of {_VALID_AGGREGATES}." - ) # Resolve survey design if provided resolved_survey, survey_weights, survey_weight_type, survey_metadata = ( @@ -949,16 +904,32 @@ def fit( att_d_p = None acrt_d_p = None - # Event study aggregation (binarized) — runs on ALL (g,t) cells - event_study_effects = None - if aggregate == "eventstudy": - event_study_effects = self._aggregate_event_study( - gt_results, - gt_bootstrap_info=gt_bootstrap_info, - unit_survey_weights=precomp.get("unit_survey_weights"), - unit_cohorts=precomp["unit_cohorts"], - anticipation=self.anticipation, - ) + # Event study aggregation (binarized) — runs on ALL (g,t) cells. + # Materialized at fit ONLY where the post-fit route cannot recompute + # it: a bootstrapped fit (n_bootstrap > 0) needs the ES targets + # inside the multiplier bootstrap (per-cell draws are not retained; + # the kit is scalars-only), so the surface is built here and RELAYED + # by results.aggregate('event_study'). Analytic fits leave it None + # and recompute from the kit on demand. FAIL-SOFT by design: any + # exception while materializing leaves the surface None (the + # bootstrap then covers the overall/dose targets only - today's + # plain-fit path) and is reported once the bootstrap outcome is + # known (see the storage rule below). + event_study_effects: Optional[Dict[int, Dict[str, Any]]] = None + _surface_failure: Optional[BaseException] = None + _bootstrap_ran = False + if self.n_bootstrap > 0: + try: + event_study_effects = self._aggregate_event_study( + gt_results, + gt_bootstrap_info=gt_bootstrap_info, + unit_survey_weights=precomp.get("unit_survey_weights"), + unit_cohorts=precomp["unit_cohorts"], + anticipation=self.anticipation, + ) + except Exception as exc: # noqa: BLE001 - fail-soft by design + _surface_failure = exc + event_study_effects = None _survey_df = None # Set by analytical branch when survey is active @@ -1047,23 +1018,46 @@ def fit( # 5. Bootstrap / Analytical SE if self.n_bootstrap > 0: - boot_result = self._run_bootstrap( - precomp, - gt_results, - gt_bootstrap_info, - post_gt, - cell_weights, - knots, - degree, - dvals, - overall_att, - overall_acrt, - agg_att_d, - agg_acrt_d, - event_study_effects, - resolved_survey=resolved_survey, - pre_unit_resolved=_unit_resolved_shared, - ) + # The engine draws the full weight matrix from the seeded + # RNG BEFORE any ES work, so the overall/dose draws are the + # same with or without the surface. Attempt 1 runs WITH the + # materialized ES surface; if it raises (anything but + # NotImplementedError, re-raised as today) the surface is + # dropped and the bootstrap is RETRIED ONCE overall-only - + # today's plain-fit path; a retry failure propagates, as + # today. On a seeded fit the retry reproduces attempt 1's + # draws exactly. + def _run_stage(_es: Optional[Dict[int, Dict[str, Any]]]) -> Dict[str, Any]: + return self._run_bootstrap( + precomp, + gt_results, + gt_bootstrap_info, + post_gt, + cell_weights, + knots, + degree, + dvals, + overall_att, + overall_acrt, + agg_att_d, + agg_acrt_d, + _es, + resolved_survey=resolved_survey, + pre_unit_resolved=_unit_resolved_shared, + ) + + if event_study_effects is not None: + try: + boot_result = _run_stage(event_study_effects) + except NotImplementedError: + raise # explicit rejections (replicate designs) propagate + except Exception as exc: # noqa: BLE001 - fail-soft by design + _surface_failure = exc + event_study_effects = None + boot_result = _run_stage(None) + else: + boot_result = _run_stage(None) + _bootstrap_ran = True att_d_se = boot_result["att_d_se"] att_d_ci_lower = boot_result["att_d_ci_lower"] att_d_ci_upper = boot_result["att_d_ci_upper"] @@ -1152,24 +1146,32 @@ def fit( acrt_d_ci_lower[idx] = ci[0] acrt_d_ci_upper[idx] = ci[1] - # Event study analytical SEs - the body lives in - # continuous_did_aggregation._compute_event_study_inference, - # shared verbatim with the post-fit aggregate('event_study') - # recompute (row M-025). fit passes its full gt_results / - # gt_bootstrap_info locals; the method reads only the - # kit-compatible key subset. - if event_study_effects is not None: - self._compute_event_study_inference( - event_study_effects, - gt_summary=gt_results, - gt_es_payload=gt_bootstrap_info, - n_units=precomp["n_units"], - unit_cohorts=precomp["unit_cohorts"], - unit_survey_weights=precomp.get("unit_survey_weights"), - unit_first_panel_row=precomp["unit_first_panel_row"], - resolved_survey=resolved_survey, - survey_df=_survey_df, - ) + # (The analytical event-study inference lives in + # continuous_did_aggregation._compute_event_study_inference + # and runs ONLY on the post-fit aggregate('event_study') + # kit recompute - analytic fits materialize no ES surface.) + + # Storage rule: the legacy field holds a surface IFF the fit carries + # bootstrap inference. When no post-treatment cells exist the + # bootstrap never runs (the overall ATT is NaN with a UserWarning), + # so an analytical surface must never sit on that bootstrapped fit + # where summary()/to_dataframe()/the DiagnosticReport would read it + # as the fit's inference. The fail-soft warning fires only when the + # bootstrap DID run but the surface was dropped. + if self.n_bootstrap > 0: + if not _bootstrap_ran: + event_study_effects = None + elif _surface_failure is not None: + warnings.warn( + f"{type(self).__name__}: the event-study/group surfaces could not " + f"be materialized on this bootstrapped fit ({_surface_failure}); " + "the bootstrap covered the overall ATT only, so " + "aggregate('event_study') / aggregate('group') are unavailable " + "here - re-fit with n_bootstrap=0 for the analytical post-fit " + "recompute.", + UserWarning, + stacklevel=2, + ) # 6. Assemble results dose_response_att = DoseResponseCurve( @@ -1253,18 +1255,15 @@ def fit( rank_deficient_action=self.rank_deficient_action, event_study_effects=event_study_effects, survey_metadata=survey_metadata, - # Per-row ES df provenance (M-092 completion): the survey df the - # ES rows' safe_inference used. None when no ES surface was - # built, on bootstrap fits (_survey_df stays None there), on - # non-survey fits, and for the replicate-undefined 0 sentinel. - event_study_df=( - float(_survey_df) - if (event_study_effects is not None and _survey_df is not None and _survey_df > 0) - else None - ), + # Per-row ES df provenance (M-092 completion): the only fit-time + # ES surface is the bootstrapped one, whose percentile inference + # used no df - always None here; the analytical post-fit + # recompute threads the kit's survey df onto its own carrier. + event_study_df=None, ) # Post-fit aggregation kit (row M-025): attached on EVERY fit; - # scalars-only on bootstrap fits (the ES route fails closed there). + # scalars-only on bootstrap fits (the ES route relays the fit-time + # surface there). results._aggregation_kit = _build_continuous_aggregation_kit( self, gt_results=gt_results, diff --git a/diff_diff/continuous_did_aggregation.py b/diff_diff/continuous_did_aggregation.py index 90860fcb..d3809305 100644 --- a/diff_diff/continuous_did_aggregation.py +++ b/diff_diff/continuous_did_aggregation.py @@ -2,9 +2,10 @@ Shared event-study aggregation machinery for ContinuousDiD (row M-025). Leaf module: hosts ``_ContinuousDiDAggregationMixin``, the binarized -event-study aggregation shared between fit-time ``fit(aggregate= -"eventstudy")`` (deprecated) and the post-fit -``ContinuousDiDResults.aggregate("event_study")`` recompute. The import +event-study aggregation shared between the fit-time surface a BOOTSTRAPPED +``fit()`` materializes (relayed by ``aggregate("event_study")``) and the +post-fit ``ContinuousDiDResults.aggregate("event_study")`` recompute on +analytic fits. The import DAG forces the split: ``continuous_did.py`` imports ``continuous_did_results.py``, so the results module cannot reach the estimator module - both instead import this leaf (numpy + diff_diff.utils @@ -108,12 +109,14 @@ def _compute_event_study_inference( """Fill analytical se/t/p/CI on the binarized event-study rows. The verbatim body of fit's former inline "Event study analytical - SEs" block: mutates ``event_study_effects`` rows in place. The - fit-time caller passes its ``gt_results`` / ``gt_bootstrap_info`` - locals; the post-fit caller passes the kit's ``gt_summary`` / - pruned ``gt_es_payload`` (key-compatible by construction). Only - runs when the fit was analytical with post-treatment cells - the - callers own that gating. + SEs" block: mutates ``event_study_effects`` rows in place. Since + 4.0 the post-fit kit recompute is the only caller (the analytical + fit-time arm was deleted - analytic fits materialize no ES surface); + it passes the kit's ``gt_summary`` / pruned ``gt_es_payload`` + (key-compatible with fit's full ``gt_results`` / + ``gt_bootstrap_info`` locals by construction). Only runs when the + fit was analytical with post-treatment cells - the caller owns + that gating. """ unit_sw = unit_survey_weights diff --git a/diff_diff/continuous_did_results.py b/diff_diff/continuous_did_results.py index 76ab0665..4121e998 100644 --- a/diff_diff/continuous_did_results.py +++ b/diff_diff/continuous_did_results.py @@ -12,7 +12,11 @@ import numpy as np import pandas as pd -from diff_diff.aggregation import AggregationMixin, AggregationResult +from diff_diff.aggregation import ( + AggregationMixin, + AggregationResult, + absent_bootstrap_surface_message, +) from diff_diff.continuous_did_aggregation import _ContinuousDiDAggregationMixin from diff_diff.results import _format_survey_block, _get_significance_stars from diff_diff.results_base import ( @@ -192,8 +196,8 @@ class ContinuousDiDResults(BaseResults, AggregationMixin): # 'simple'/'dose' are views. _aggregation_kit: Optional[Any] = field(default=None, repr=False, compare=False) # Scalar survey df governing the event-study rows' t-inference. None on - # non-survey fits, on bootstrapped fits (percentile inference; the ES - # recompute also fails closed there), when no event-study surface was + # non-survey fits, on bootstrapped fits (percentile inference; the + # relayed surface carries no df), when no event-study surface was # built, and for the replicate-undefined 0 sentinel. Appended last per # the positional-__init__ convention above. event_study_df: Optional[float] = None @@ -503,12 +507,12 @@ def to_dataframe(self, level: str = "dose_response") -> pd.DataFrame: elif level == "event_study": if self.event_study_effects is None: raise ValueError( - "Event study effects not computed. Call " + "Event study effects not stored (only bootstrapped fits " + "materialize the surface at fit). Call " "results.aggregate('event_study') for the unified " - "post-fit container (on a bootstrapped fit, re-fit " - "with n_bootstrap=0 or use the deprecated fit-time " - "aggregate='eventstudy'); a result unpickled from an " - "older release must be re-fit with diff-diff >= 3.9." + "post-fit container - analytic fits recompute it from " + "the retained kit; a result unpickled from an older " + "release must be re-fit with diff-diff >= 3.9." ) rows = [] for rel_t, data in sorted(self.event_study_effects.items()): @@ -544,9 +548,11 @@ def significance_stars(self) -> str: # MIXED architecture: 'simple' and 'dose' are pure VIEWS over stored # public fields (the dCDH precedent - nothing recomputed, so they # work on ANY fit including bootstrap fits and legacy pickles, - # relaying the stored inference verbatim); 'event_study' is a KIT - # RECOMPUTE (the EfficientDiD class) from the pruned per-cell IF - # payload, failing closed on bootstrap fits. + # relaying the stored inference verbatim); 'event_study' is a HYBRID: + # a KIT RECOMPUTE (the EfficientDiD class) from the pruned per-cell IF + # payload on analytic fits, and a RELAY of the percentile surface + # fit() materialized on bootstrapped fits (the kit is scalars-only + # there; balance_e is rejected by the mixin for this producer). # ------------------------------------------------------------------ def _stored_inference_df(self) -> float: @@ -675,16 +681,27 @@ def _aggregate_compute( ) bk = kit.bookkeeping if bk["n_bootstrap"] > 0: - raise NotImplementedError( - "aggregate('event_study') on a bootstrapped ContinuousDiD " - "fit is not implemented - the fit-time event study used " - "multiplier-bootstrap inference whose per-cell draws are " - "not retained, and an analytical recompute would silently " - "differ. Until 4.0 the deprecated fit-time " - "aggregate='eventstudy' still computes the bootstrap " - "surface, or re-fit with n_bootstrap=0 for the analytical " - "post-fit route." + # 4.0 PR 3c: the kit is scalars-only on every bootstrapped fit + # (no recompute is possible), so RELAY the percentile surface + # fit() materialized, with kit-snapshot provenance. Gate on the + # KIT scalar, not the mutable public field (no new kit key: a + # 3.9-3.13 pickle must not KeyError). A bootstrapped fit storing + # no surface (pre-4.0 pickle, fail-soft drop, or the + # no-post-cells fit whose bootstrap never ran) raises. + if self.event_study_effects is None: + raise ValueError( + absent_bootstrap_surface_message(type(self).__name__, "event_study") + ) + meta_b = bk["survey_metadata"] + carrier_b = dataclasses.replace( + self, + survey_metadata=(dataclasses.replace(meta_b) if meta_b is not None else None), + alpha=kit.alpha, + anticipation=kit.anticipation, + base_period=bk["base_period"], + event_study_df=None, # percentile inference governs no df ) + return build_event_study_surface(carrier_b) host = _ContinuousKitAggregator(alpha=kit.alpha) es = host._aggregate_event_study( bk["gt_summary"], diff --git a/diff_diff/diagnostic_report.py b/diff_diff/diagnostic_report.py index 790d8f80..a35a1fbb 100644 --- a/diff_diff/diagnostic_report.py +++ b/diff_diff/diagnostic_report.py @@ -12,14 +12,15 @@ not a traffic-light enum. See ``docs/methodology/REPORTING.md``. - No estimator fitting. Every effect, SE, p-value, CI, and sensitivity bound is either read from ``results``, derived from the result's own - post-fit ``aggregate('event_study')`` surface (a view or retained-kit - recompute — for ImputationDiD a panel-backed recompute, for TwoStageDiD - a fresh Stage-2 OLS + GMM sandwich over the retained frame; used only - when the raw ``event_study_effects`` field is absent, and failing - closed to an explicit skip on kit-less/legacy-pickle fits, the sibling - estimators' bootstrap gates, and backend-mismatched CS bootstrap - replays — bootstrapped CS fits themselves derive successfully via the - percentile-bootstrap replay), or + post-fit ``aggregate('event_study')`` surface (a view, a retained-kit + recompute, or a relay of the surface a bootstrapped fit materialized - + for analytical ImputationDiD a panel-backed recompute, for analytical + TwoStageDiD a fresh Stage-2 OLS + GMM sandwich over the retained frame, + for every CallawaySantAnna / EfficientDiD fit the kit recompute or the + percentile-bootstrap replay; used only when the raw + ``event_study_effects`` field is absent, and failing closed to an + explicit skip on kit-less/legacy-pickle fits, fail-soft-cleared + bootstrapped surfaces, and backend-mismatched CS bootstrap replays), or produced by an existing diff-diff utility. May call ``check_parallel_trends`` / ``BaconDecomposition`` / ``EfficientDiD.hausman_pretest`` when the caller supplies the panel + @@ -806,12 +807,14 @@ def _resolve_event_study_surface( authoritative). The raw ``event_study_effects`` field always takes precedence when it - is not ``None`` — including the requested-but-empty ``{}`` sentinel, - which encodes fit-time configuration (e.g. an ImputationDiD - ``balance_e=`` that emptied the window) that a re-derivation without - those settings would silently overturn. Derivation failures - (bootstrapped fits, missing kits) are cached as skip reasons, never - raised; warnings emitted by a kit recompute are captured into + is not ``None`` — including the materialized-but-empty ``{}`` + sentinel. Since 4.0 the raw field is populated only by producers + that materialize the surface at fit (StackedDiD, SunAbraham, + bootstrapped ImputationDiD / TwoStageDiD / ContinuousDiD - always + with ``balance_e=None``), so an empty dict records that the fit + itself found no estimable horizon, which a re-derivation would + silently overturn. Derivation failures (missing kits, cleared + bootstrapped surfaces) are cached as skip reasons, never raised; warnings emitted by a kit recompute are captured into ``self._derived_es_warnings`` and re-published by the consuming runners (record-and-republish, mirroring the sensitivity helper). """ @@ -827,9 +830,8 @@ def _resolve_event_study_surface( why: Optional[str] = None if getattr(r, "event_study_effects", None) is not None: # Raw precedence, ``is not None`` deliberately: ``{}`` is the - # REQUESTED-but-no-estimable-horizons sentinel and must never be - # silently re-derived (the fit-time balance_e/aggregate settings - # behind it are not retained for replay). + # materialized-but-no-estimable-horizons sentinel (a fit-time + # producer found nothing) and must never be silently re-derived. pass elif name == "WooldridgeDiDResults": # Its bespoke aggregate() MUTATES the results object in place; @@ -889,11 +891,12 @@ def _resolve_event_study_surface( surface = candidate surface_dict = _surface_to_event_study_dict(candidate) except Exception as exc: # noqa: BLE001 — fail-soft by design: - # expected failures are NotImplementedError (the sibling - # estimators' bootstrap gates, pretrends+replicate, CS - # legacy-pickle/backend-mismatch replay refusals — a - # bootstrapped CS fit itself now derives via the replay) - # and ValueError (missing kit), + # expected failures are NotImplementedError + # (pretrends+replicate, CS legacy-pickle/backend-mismatch + # replay refusals — a bootstrapped CS fit itself derives via + # the replay) and ValueError (missing kit; a bootstrapped + # Imputation/TwoStage/Continuous fit whose fail-soft + # materialization left no surface to relay), # but the surface builder can raise bare TypeError and this # resolver runs on the applicable_checks path with no outer # guard; an escaped exception would hard-fail the report. @@ -994,9 +997,9 @@ def _instance_skip_reason(self, check: str) -> Optional[str]: if not pre_coefs and n_dropped_undefined == 0: # SpilloverDiD's event-study switch is the # ``SpilloverDiD(..., event_study=True)`` constructor - # kwarg, not the ``aggregate='event_study'`` argument - # the generic staggered-estimator message points at - # (SpilloverDiD has no ``aggregate`` kwarg). Emit an + # kwarg, not the post-fit ``aggregate('event_study')`` + # route the generic staggered-estimator message points + # at (SpilloverDiD has no post-fit aggregation). Emit an # estimator-accurate remediation for this family. if name == "SpilloverDiDResults": return ( diff --git a/diff_diff/guides/llms-full.txt b/diff_diff/guides/llms-full.txt index 94fb6fb8..af6e4be6 100644 --- a/diff_diff/guides/llms-full.txt +++ b/diff_diff/guides/llms-full.txt @@ -458,8 +458,6 @@ imp.fit( time: str, first_treat: str, covariates: list[str] = None, - aggregate: Any = NOT_SUPPLIED, # DEPRECATED (M-021): warns, still works - use results.aggregate() post-fit - balance_e: Any = NOT_SUPPLIED, # DEPRECATED (M-118): moves onto results.aggregate(balance_e=) survey_design: SurveyDesign = None, # Optional design-based inference (pweight + analytical strata/PSU/FPC or replicate BRR/Fay/JK1/JKn/SDR) ) -> ImputationDiDResults ``` @@ -473,8 +471,10 @@ est = ImputationDiD() results = est.fit(data, outcome='outcome', unit='unit', time='period', first_treat='first_treat') results.print_summary() -es = results.aggregate('event_study') # post-fit (M-021); balance_e= lives here too +es = results.aggregate('event_study') # post-fit (M-021); balance_e= lives here (analytical fits) plot_event_study(es) +# n_bootstrap > 0: the percentile ES/group surfaces are computed at fit and +# aggregate('event_study') / aggregate('group') relay them (balance_e unavailable). ``` ### TwoStageDiD @@ -507,8 +507,6 @@ ts.fit( time: str, first_treat: str, covariates: list[str] = None, - aggregate: Any = NOT_SUPPLIED, # DEPRECATED (M-022): warns, still works - use results.aggregate() post-fit - balance_e: Any = NOT_SUPPLIED, # DEPRECATED (M-119): moves onto results.aggregate(balance_e=) ) -> TwoStageDiDResults ``` @@ -521,7 +519,9 @@ est = TwoStageDiD() results = est.fit(data, outcome='outcome', unit='unit', time='period', first_treat='first_treat') results.print_summary() -es = results.aggregate('event_study') # post-fit (M-022); balance_e= lives here too +es = results.aggregate('event_study') # post-fit (M-022); balance_e= lives here (analytical fits) +# n_bootstrap > 0: the percentile ES/group surfaces are computed at fit and +# aggregate('event_study') / aggregate('group') relay them (balance_e unavailable). ``` ### SpilloverDiD @@ -786,10 +786,6 @@ cdid.fit( time: str, first_treat: str, dose: str, # Column with continuous treatment dose - aggregate: Any = NOT_SUPPLIED, # DEPRECATED (M-025) - removed in 4.0; use post-fit results.aggregate(); - # legacy values None/"dose"/"eventstudy" still route until 4.0 (any - # supplied value warns; "dose" was always a no-op - the curves are - # always computed) covariates: list[str] = None, # Conditional parallel trends (X). None = unconditional ) -> ContinuousDiDResults ``` @@ -805,12 +801,10 @@ results = est.fit(data, outcome='outcome', unit='unit', time='period', results.print_summary() dose_table = results.aggregate('dose') # ATT(d) + ACRT(d) rows (view; any fit) overall = results.aggregate('simple') # att + acrt rows (view; any fit) -# Binarized event study: analytical fits only (bootstrapped fits raise - -# re-fit with n_bootstrap=0, or the deprecated fit-time aggregate='eventstudy' -# until 4.0): -es = ContinuousDiD(seed=42).fit(data, outcome='outcome', unit='unit', - time='period', first_treat='first_treat', - dose='dose').aggregate('event_study') +# Binarized event study: an analytical fit recomputes it from the retained +# kit; a bootstrapped fit (n_bootstrap > 0) computes the percentile surface +# at fit and aggregate('event_study') relays it (4.0): +es = results.aggregate('event_study') ``` ### HeterogeneousAdoptionDiD @@ -2109,9 +2103,9 @@ ImputationDiDResults, TwoStageDiDResults, StackedDiDResults, and EfficientDiDRes | `overall_t_stat` | `float` | T-statistic | | `overall_p_value` | `float` | P-value | | `overall_conf_int` | `tuple[float, float]` | CI | -| `event_study_effects` | `dict[int, dict] | None` | Legacy field. `None` on every EfficientDiD fit since 4.0 (PR 3b) and on plain ImputationDiD/TwoStageDiD fits that omit `aggregate=`; ImputationDiD/TwoStageDiD still populate it via the deprecated fit-time `aggregate=` (removal pending, PR 3c); StackedDiD always populates it (row M-024). The canonical surface is the `aggregate('event_study')` CONTAINER | -| `event_study_df` | `float | dict | None` | Producer-side per-row df channel for the unified event-study container. On `EfficientDiDResults`/`ImputationDiDResults` (3.12, M-092 completion): the scalar survey df governing the ES rows' t-inference (ImputationDiD threads the FINAL replicate-override value; leads included on survey fits); `None` on non-survey fits, on bootstrapped fits (percentile inference used no df), when no fit-time ES surface was built, and for the replicate-undefined `0` sentinel | -| `group_effects` | `dict | None` | Legacy field. `None` on every EfficientDiD fit since 4.0 (PR 3b) and on plain ImputationDiD/TwoStageDiD fits that omit `aggregate=`; ImputationDiD/TwoStageDiD still populate it via the deprecated fit-time `aggregate=` (removal pending, PR 3c). The canonical surface is the `aggregate('group')` AggregationResult container | +| `event_study_effects` | `dict[int, dict] | None` | Legacy field. `None` on analytical fits and on every EfficientDiD fit; populated on bootstrapped ImputationDiD/TwoStageDiD fits (the percentile surface computed at fit, relayed by `aggregate('event_study')`); StackedDiD always populates it (row M-024). The canonical surface is the `aggregate('event_study')` CONTAINER | +| `event_study_df` | `float | dict | None` | Producer-side per-row df channel for the unified event-study container. On `EfficientDiDResults`/`ImputationDiDResults` (3.12, M-092 completion): the scalar survey df governing the ES rows' t-inference (ImputationDiD threads the FINAL replicate-override value; leads included on survey fits); `None` on non-survey fits, on bootstrapped fits (percentile inference used no df), on analytical fits (no surface is stored - the kit recomputes), and for the replicate-undefined `0` sentinel | +| `group_effects` | `dict | None` | Legacy field. `None` on analytical fits and on every EfficientDiD fit; populated on bootstrapped ImputationDiD/TwoStageDiD fits (relayed by `aggregate('group')`). The canonical surface is the `aggregate('group')` AggregationResult container | | `groups` | `list` | Treatment cohorts | | `time_periods` | `list` | All time periods | | `n_obs` | `int` | Number of observations | @@ -2129,7 +2123,7 @@ Each event study effect dict contains: `effect`, `se`, `t_stat`, `p_value`, `con | `n_clusters` | `int | None` | Number of effective clusters; `None` under survey designs and `None` under EfficientDiD's default unclustered fit | | `df_convention` | `str | None` | On the knob-carrying containers (`StackedDiDResults`, `ImputationDiDResults`, `WooldridgeDiDResults`, `LPDiDResults`, `SunAbrahamResults`): the configured df convention (3.9 / M-127); `StackedDiDResults` additionally carries `inference_df` (the overall-ATT df actually used) | -**Methods:** `summary()`, `print_summary()`, `to_dataframe()`, `to_dict()` (flat dict of headline aliases + `vcov_type` + conditional `cluster_name`/`n_clusters`/`n_bootstrap`/`inference_method`); `aggregate(type, weights=None, *, balance_e=None)` on the shipped post-fit adopters (`StackedDiDResults` views since 3.9/M-024; `EfficientDiDResults` recomputes from retained EIFs since 3.9/M-023 - on bootstrapped fits its recompute levels REPLAY the fit-time multiplier bootstrap (percentile inference; M-023); `ImputationDiDResults` and `TwoStageDiDResults` recompute from their PANEL-BACKED kits since 3.9/M-021/M-022 - on bootstrapped fits their recompute levels raise while `'simple'` and, where supported, `'total'` relay the stored bootstrap inference with a NaN df column (the per-level rule, converged with M-027); `'total'` (3.10) is the estimator-owned total incremental outcome on `CallawaySantAnnaResults`/`EfficientDiDResults`/`ImputationDiDResults`/`TwoStageDiDResults` - panel non-survey fits only, fails closed with the reason elsewhere; `ContinuousDiDResults` is MIXED since 3.9/M-025 - `'simple'`/`'dose'` are views over stored fields that work on any fit incl. bootstrapped, `'event_study'` recomputes from a pruned per-cell IF kit and raises on bootstrapped fits; `HeterogeneousAdoptionDiDResults` and `HeterogeneousAdoptionDiDEventStudyResults` are pure views since 3.9/M-027 - `'simple'` on the overall class, `'event_study'` on the event-study class, no kit, work on pickles from any release) +**Methods:** `summary()`, `print_summary()`, `to_dataframe()`, `to_dict()` (flat dict of headline aliases + `vcov_type` + conditional `cluster_name`/`n_clusters`/`n_bootstrap`/`inference_method`); `aggregate(type, weights=None, *, balance_e=None)` on the shipped post-fit adopters (`StackedDiDResults` views since 3.9/M-024; `EfficientDiDResults` recomputes from retained EIFs since 3.9/M-023 - on bootstrapped fits its recompute levels REPLAY the fit-time multiplier bootstrap (percentile inference; M-023); `ImputationDiDResults` and `TwoStageDiDResults` recompute from their PANEL-BACKED kits since 3.9/M-021/M-022 on analytical fits - on bootstrapped fits the ES/group percentile surfaces are computed at fit and `'event_study'`/`'group'` RELAY them (4.0; `balance_e=` unavailable there) while `'simple'` and, where supported, `'total'` relay the stored bootstrap inference with a NaN df column (the per-level rule, converged with M-027); `'total'` (3.10) is the estimator-owned total incremental outcome on `CallawaySantAnnaResults`/`EfficientDiDResults`/`ImputationDiDResults`/`TwoStageDiDResults` - panel non-survey fits only, fails closed with the reason elsewhere; `ContinuousDiDResults` is MIXED since 3.9/M-025 - `'simple'`/`'dose'` are views over stored fields that work on any fit incl. bootstrapped, `'event_study'` recomputes from a pruned per-cell IF kit on analytical fits and relays the percentile surface computed at fit on bootstrapped fits (4.0); `HeterogeneousAdoptionDiDResults` and `HeterogeneousAdoptionDiDEventStudyResults` are pure views since 3.9/M-027 - `'simple'` on the overall class, `'event_study'` on the event-study class, no kit, work on pickles from any release) ### ContinuousDiDResults @@ -2154,8 +2148,8 @@ Each event study effect dict contains: `effect`, `se`, `t_stat`, `p_value`, `con | `n_obs` | `int` | Number of observations | | `n_treated_units` | `int` | Treated units | | `n_control_units` | `int` | Control units | -| `event_study_effects` | `dict[int, dict] or None` | Event study effects (populated only by the DEPRECATED fit-time `aggregate="eventstudy"`; the post-fit successor `results.aggregate('event_study')` returns a unified `EventStudyResults` container instead of populating this field) | -| `event_study_df` | `float | None` | Scalar survey df governing the ES rows' t-inference (3.12, M-092 completion); `None` on non-survey fits, on bootstrapped fits, when no fit-time ES surface was built, and for the replicate-undefined `0` sentinel | +| `event_study_effects` | `dict[int, dict] or None` | Legacy field - `None` on analytical fits; on bootstrapped fits (`n_bootstrap > 0`) the percentile event study computed at fit, which `results.aggregate('event_study')` relays as a unified `EventStudyResults` container | +| `event_study_df` | `float | None` | Scalar survey df governing the ES rows' t-inference (3.12, M-092 completion); `None` on non-survey fits, on bootstrapped fits, on analytical fits (no surface stored), and for the replicate-undefined `0` sentinel | **DoseResponseCurve** sub-dataclass: @@ -2168,7 +2162,7 @@ Each event study effect dict contains: `effect`, `se`, `t_stat`, `p_value`, `con | `conf_int_upper` | `np.ndarray` | Upper CI bound | | `target` | `str` | `"att"` or `"acrt"` | -**Methods:** `summary()`, `print_summary()`, `to_dataframe()`, `aggregate(type, ...)` (M-025 MIXED adopter: `'simple'` = 2-row att+acrt view, `'dose'` = 2N-row target-discriminated curve view - both work on any fit; `'event_study'` = kit recompute, analytical fits only) +**Methods:** `summary()`, `print_summary()`, `to_dataframe()`, `aggregate(type, ...)` (M-025 MIXED adopter: `'simple'` = 2-row att+acrt view, `'dose'` = 2N-row target-discriminated curve view - both work on any fit; `'event_study'` = kit recompute on analytical fits, relay of the percentile surface computed at fit on bootstrapped fits - 4.0) ### HeterogeneousAdoptionDiDResults diff --git a/diff_diff/guides/llms-practitioner.txt b/diff_diff/guides/llms-practitioner.txt index acacae4e..c026aaa0 100644 --- a/diff_diff/guides/llms-practitioner.txt +++ b/diff_diff/guides/llms-practitioner.txt @@ -484,8 +484,7 @@ Aggregate treatment effects may mask important variation. ### For CallawaySantAnna, EfficientDiD, ImputationDiD, TwoStageDiD (post-fit `aggregate()`) ```python # Post-fit, no refit needed (the fit-time aggregate= / balance_e= keywords -# were removed at 4.0 on CS/StackedDiD/EfficientDiD and remain deprecated on -# ImputationDiD/TwoStageDiD until their removal): +# were removed at 4.0 on CS/StackedDiD/EfficientDiD/ImputationDiD/TwoStageDiD): results = cs.fit(data, ...) print(results.overall_att) # Overall ATT print(results.aggregate('group').to_dataframe()) # Per-cohort ATTs @@ -497,23 +496,25 @@ print(results.aggregate('event_study', balance_e=2).to_dataframe()) print(results.aggregate('total').to_dataframe()) # BOOTSTRAPPED fits: the CS, DMLDiD, and EfficientDiD recompute levels # (event_study/group) REPLAY the fit-time multiplier bootstrap post-fit -# (percentile inference; no refit needed) — but they still RAISE on -# ImputationDiD/TwoStageDiD, where aggregate('simple') and, where -# supported, aggregate('total') relay the stored bootstrap inference and -# the deprecated fit-time aggregation remains the ES/group route: -results = imp.fit(data, ..., aggregate='all') # ImputationDiD/TwoStageDiD only +# (percentile inference; no refit needed). ImputationDiD/TwoStageDiD compute +# the percentile ES/group surfaces AT FIT when n_bootstrap > 0 and +# aggregate('event_study') / aggregate('group') RELAY them (balance_e= is +# unavailable there - re-fit with n_bootstrap=0 for a balanced window); +# aggregate('simple') / aggregate('total') relay the stored inference: +results = imp.fit(data, ...) # ImputationDiD(n_bootstrap=199) +print(results.aggregate('event_study').to_dataframe()) # relayed percentile surface ``` ### For ContinuousDiD (MIXED post-fit `aggregate()`, row M-025) ```python # Dose-response curves + overall ATT/ACRT are ALWAYS computed by fit(); # 'dose' and 'simple' are views that work on ANY fit (bootstrapped too): -results = cdid.fit(data, ...) # no aggregate= kwarg (deprecated) +results = cdid.fit(data, ...) # no aggregate= kwarg (removed at 4.0) print(results.aggregate('dose').to_dataframe()) # ATT(d) + ACRT(d) rows print(results.aggregate('simple').to_dataframe()) # att + acrt overall rows -# The binarized event study RECOMPUTES from the retained kit - analytical -# fits only (a bootstrapped fit raises; re-fit with n_bootstrap=0 or use -# the deprecated fit-time aggregate='eventstudy' until 4.0): +# The binarized event study RECOMPUTES from the retained kit on analytical +# fits; a bootstrapped fit (n_bootstrap > 0) computes the percentile event +# study at fit and aggregate('event_study') relays it (4.0): print(results.aggregate('event_study').to_dataframe()) ``` diff --git a/diff_diff/guides/llms.txt b/diff_diff/guides/llms.txt index 813a3aa8..34543103 100644 --- a/diff_diff/guides/llms.txt +++ b/diff_diff/guides/llms.txt @@ -21,7 +21,7 @@ diagnostic steps produces unreliable results. 4. **Choose estimator** — staggered adoption → CS/SA/BJS (NOT plain TWFE); few treated units → SDiD; factor confounding → TROP; simple 2x2 → DiD. Run `BaconDecomposition` to diagnose TWFE bias. 5. **Estimate** — `estimator.fit(data, ...)`. Always print the cluster count first and choose inference method based on the result (cluster-robust if >= 50 clusters, wild bootstrap if fewer — for DifferenceInDifferences pass `cluster=`; TwoWayFixedEffects auto-clusters at unit level). 6. **Sensitivity analysis** — `compute_honest_did(results)` for bounds under PT violations (MultiPeriodDiD, CS, or dCDH natively; the TwoWayFixedEffects `event_study=True` surface and a StackedDiD `results.aggregate('event_study')` container also admit - Stacked needs `kappa_pre >= 2` so estimated pre-periods exist), `run_all_placebo_tests()` for 2x2 falsification, specification comparisons for staggered designs. -7. **Heterogeneity** — CS: `results.aggregate('group')`/`.aggregate('event_study')` post-fit, no refit (the fit-time `aggregate=`/`balance_e=` keywords were removed at 4.0 and `results.event_study_effects`/`group_effects` are None on every CS fit; `compute_honest_did` / `compute_pretrends_power` / `plot_event_study` all accept the post-fit `results.aggregate('event_study')` container directly; BOOTSTRAPPED CS fits included: the recompute levels `'event_study'`/`'group'` REPLAY the fit-time multiplier bootstrap from the kit's retained RNG state (percentile inference matching a fit-time aggregation; the container's vcov is the covariance of the bootstrap draws, so its diagonal is the bootstrap se**2), while `.aggregate('simple')` and, where supported, `.aggregate('total')` relay the stored bootstrap inference (NaN df column); only pre-replay legacy pickles and cross-weight-backend artifacts fail closed with a refit message). NEW in 3.10: `.aggregate('total')` on CS/EfficientDiD/ImputationDiD/TwoStageDiD - the estimator-owned total incremental outcome (exact relay C x overall over the finite-masked complete-case support; single target='total' row; bootstrap-safe RELAY; panel non-survey fits only - repeated-cross-section and declared survey_design fits raise NotImplementedError with the reason); dCDH: `results.aggregate('event_study')`/`.aggregate('simple')` post-fit views (bootstrap fits included — pure views); SA: `results.event_study_effects`/`to_dataframe(level='cohort')`; StackedDiD: `results.aggregate('event_study')`/`.aggregate('simple')` post-fit views (the surface is ALWAYS computed at fit since 3.9 - row M-024 - and the container admits into `compute_honest_did`/`compute_pretrends_power` with `kappa_pre >= 2`); EDiD: `results.aggregate('event_study')`/`.aggregate('group')`/`.aggregate('simple')` post-fit, RECOMPUTED from retained EIFs (3.9, row M-023; the fit-time keywords were removed at 4.0; on bootstrapped EDiD fits the recompute levels REPLAY the fit-time multiplier bootstrap from the kit's retained RNG state (percentile inference matching a fit-time aggregation; only pre-replay legacy pickles and cross-weight-backend artifacts fail closed with a refit message) while `.aggregate('simple')` relays the stored bootstrap inference; EDiD containers are NOT admitted into honest/pretrends - no joint ES covariance); BJS/TwoStageDiD: `results.aggregate('event_study')`/`.aggregate('group')`/`.aggregate('simple')` post-fit on ImputationDiD and TwoStageDiD too (3.9, rows M-021/M-022; recomputed from panel-backed kits, `balance_e=` on `aggregate('event_study')`; on bootstrapped fits the recompute levels raise while `.aggregate('simple')` relays the stored bootstrap inference - use the deprecated fit-time aggregation for a bootstrapped ES/group surface; their containers are not admitted into honest/pretrends - Imputation by design, TwoStage deferred pending a normalization derivation); CGBS continuous: ContinuousDiD is a MIXED adopter (3.9, row M-025) - `results.aggregate('dose')` (ATT(d)+ACRT(d) rows) and `.aggregate('simple')` (att+acrt rows) are views over the always-computed curves and work on ANY fit incl. bootstrapped, while `.aggregate('event_study')` recomputes the binarized event study from a pruned per-cell IF kit and raises on bootstrapped fits (use the deprecated fit-time `aggregate='eventstudy'` there until 4.0; its container is not admitted into honest/pretrends - no joint ES covariance and no reference normalization); HAD: `results.aggregate('simple')` (overall two-period fits; the target column carries the WAS estimand label) / `.aggregate('event_study')` (multi-period fits) - pure views, work on any fit (3.9, rows M-027/M-139; fit() selects the mode from the panel shape; HAD containers are not admitted into honest/pretrends - no joint cross-horizon covariance, deferred); subgroup re-estimation. +7. **Heterogeneity** — CS: `results.aggregate('group')`/`.aggregate('event_study')` post-fit, no refit (the fit-time `aggregate=`/`balance_e=` keywords were removed at 4.0 and `results.event_study_effects`/`group_effects` are None on every CS fit; `compute_honest_did` / `compute_pretrends_power` / `plot_event_study` all accept the post-fit `results.aggregate('event_study')` container directly; BOOTSTRAPPED CS fits included: the recompute levels `'event_study'`/`'group'` REPLAY the fit-time multiplier bootstrap from the kit's retained RNG state (percentile inference matching a fit-time aggregation; the container's vcov is the covariance of the bootstrap draws, so its diagonal is the bootstrap se**2), while `.aggregate('simple')` and, where supported, `.aggregate('total')` relay the stored bootstrap inference (NaN df column); only pre-replay legacy pickles and cross-weight-backend artifacts fail closed with a refit message). NEW in 3.10: `.aggregate('total')` on CS/EfficientDiD/ImputationDiD/TwoStageDiD - the estimator-owned total incremental outcome (exact relay C x overall over the finite-masked complete-case support; single target='total' row; bootstrap-safe RELAY; panel non-survey fits only - repeated-cross-section and declared survey_design fits raise NotImplementedError with the reason); dCDH: `results.aggregate('event_study')`/`.aggregate('simple')` post-fit views (bootstrap fits included — pure views); SA: `results.event_study_effects`/`to_dataframe(level='cohort')`; StackedDiD: `results.aggregate('event_study')`/`.aggregate('simple')` post-fit views (the surface is ALWAYS computed at fit since 3.9 - row M-024 - and the container admits into `compute_honest_did`/`compute_pretrends_power` with `kappa_pre >= 2`); EDiD: `results.aggregate('event_study')`/`.aggregate('group')`/`.aggregate('simple')` post-fit, RECOMPUTED from retained EIFs (3.9, row M-023; the fit-time keywords were removed at 4.0; on bootstrapped EDiD fits the recompute levels REPLAY the fit-time multiplier bootstrap from the kit's retained RNG state (percentile inference matching a fit-time aggregation; only pre-replay legacy pickles and cross-weight-backend artifacts fail closed with a refit message) while `.aggregate('simple')` relays the stored bootstrap inference; EDiD containers are NOT admitted into honest/pretrends - no joint ES covariance); BJS/TwoStageDiD: `results.aggregate('event_study')`/`.aggregate('group')`/`.aggregate('simple')` post-fit on ImputationDiD and TwoStageDiD too (3.9, rows M-021/M-022; recomputed from panel-backed kits, `balance_e=` on `aggregate('event_study')`; on bootstrapped fits the percentile ES/group surfaces are computed at fit and `.aggregate('event_study')`/`.aggregate('group')` RELAY them (4.0; `balance_e=` unavailable there) while `.aggregate('simple')` relays the stored bootstrap inference; their containers are not admitted into honest/pretrends - Imputation by design, TwoStage deferred pending a normalization derivation); CGBS continuous: ContinuousDiD is a MIXED adopter (3.9, row M-025) - `results.aggregate('dose')` (ATT(d)+ACRT(d) rows) and `.aggregate('simple')` (att+acrt rows) are views over the always-computed curves and work on ANY fit incl. bootstrapped, while `.aggregate('event_study')` recomputes the binarized event study from a pruned per-cell IF kit on analytical fits and relays the percentile surface computed at fit on bootstrapped fits (4.0; its container is not admitted into honest/pretrends - no joint ES covariance and no reference normalization); HAD: `results.aggregate('simple')` (overall two-period fits; the target column carries the WAS estimand label) / `.aggregate('event_study')` (multi-period fits) - pure views, work on any fit (3.9, rows M-027/M-139; fit() selects the mode from the panel shape; HAD containers are not admitted into honest/pretrends - no joint cross-horizon covariance, deferred); subgroup re-estimation. 8. **Robustness** — compare 2-3 estimators (CS vs SA vs BJS), MUST report with and without covariates (shows whether conditioning drives identification), present pre-trends and sensitivity bounds. After estimation, call `practitioner_next_steps(results)` for context-aware diff --git a/diff_diff/imputation.py b/diff_diff/imputation.py index 41b654a4..71be2ce7 100644 --- a/diff_diff/imputation.py +++ b/diff_diff/imputation.py @@ -32,7 +32,6 @@ from scipy import stats from diff_diff._base import BaseEstimator -from diff_diff._deprecation import NOT_SUPPLIED from diff_diff.aggregation import AggregationKit from diff_diff.imputation_aggregation import ( # noqa: F401 (compat re-exports) _compute_target_weights, @@ -152,8 +151,7 @@ class ImputationDiD(ImputationDiDBootstrapMixin, _ImputationAggregationMixin, Ba df_convention : {"residual", "cluster", "normal"}, default "residual" Degrees-of-freedom convention for the PRETRENDS lead regression's per-lead t/p/CI — the one ImputationDiD surface running the shared - clustered CR1 sandwich (``pretrends=True``, surfaced fit-time via - the deprecated ``aggregate="event_study"``/``"all"`` or post-fit via + clustered CR1 sandwich (``pretrends=True``, surfaced via ``results.aggregate('event_study')``). ``"residual"`` (default) uses the lead regression's residual df (``n − k_kept − absorbed [time, unit] rank``) — the 3.9 fix: previously silent normal-theory @@ -271,8 +269,6 @@ def fit( time: str, first_treat: str, covariates: Optional[List[str]] = None, - aggregate: Any = NOT_SUPPLIED, - balance_e: Any = NOT_SUPPLIED, survey_design: Optional["SurveyDesign"] = None, ) -> ImputationDiDResults: """ @@ -293,20 +289,6 @@ def fit( Use 0 (or np.inf) for never-treated units. covariates : list of str, optional List of covariate column names. - aggregate : str, optional - DEPRECATED (3.9, removed in 4.0; row M-021): aggregate as a - post-fit step instead — ``results.aggregate('event_study')`` / - ``.aggregate('group')`` / ``.aggregate('simple')`` / - ``.aggregate('total')``. Supplying - ANY value (``None`` included) warns; the deprecated path still - works and returns exactly the numbers it always did - (fit-time mode: None/"simple" overall only, "event_study", - "group", or "all"). - balance_e : int, optional - DEPRECATED (3.9, removed in 4.0; row M-118): moves onto - ``results.aggregate('event_study', balance_e=...)``. Restricts - the event study to cohorts observed at every relative time in - ``[-balance_e, max_h]`` (the balanced-window rule). survey_design : SurveyDesign, optional Survey design specification for design-based inference. Supports pweight only (aweight/fweight raise ValueError). Supports strata, @@ -324,38 +306,6 @@ def fit( ValueError If required columns are missing or data validation fails. """ - # M-021/M-118 deprecation shim (CS-style joint warning): a plain - # fit() never warns; supplying EITHER param with ANY value (None - # included) warns once, then the legacy routing below runs - # unchanged - the deprecated path returns exactly the numbers it - # always did (no new value validation; unknown strings still act - # like None). The post-fit successor validates its own vocabulary. - # The sentinel is normalized HERE, before every downstream read of - # ``aggregate`` (the pretrends+replicate gate and the df_convention - # reachability warning below both read it). - _deprecated_passed = [ - n - for n, v in (("aggregate", aggregate), ("balance_e", balance_e)) - if v is not NOT_SUPPLIED - ] - if _deprecated_passed: - _args = " / ".join(f"{n}=" for n in _deprecated_passed) - warnings.warn( - f"ImputationDiD.fit({_args}) is deprecated and will be " - "removed in 4.0. Fit once, then aggregate as a post-fit " - "step: results = ImputationDiD().fit(...); " - "results.aggregate('event_study') / .aggregate('group') / " - ".aggregate('simple') / .aggregate('total'). balance_e moves onto aggregate() " - "alongside it: results.aggregate('event_study', " - "balance_e=2).", - FutureWarning, - stacklevel=2, - ) - if aggregate is NOT_SUPPLIED: - aggregate = None - if balance_e is NOT_SUPPLIED: - balance_e = None - # Re-validate vcov_type at fit-time: set_params validates eagerly # (BaseEstimator probe re-init), so this only catches DIRECT # attribute mutation (est.vcov_type = ...). @@ -375,54 +325,32 @@ def fit( if missing: raise ValueError(f"Missing columns: {missing}") - # pretrends + analytical survey is supported (Phase 8e-iii). - # Replicate-weight surveys need per-replicate lead regression refits - # which are not yet implemented — reject that combination. - if ( - self.pretrends - and survey_design is not None - and survey_design.replicate_method is not None - and aggregate in ("event_study", "all") - ): - raise NotImplementedError( - "pretrends=True is not yet compatible with replicate-weight " - "survey designs. Analytical survey designs (strata/PSU/FPC) " - "are supported. Use pretrends=False with replicate weights." - ) + # (pretrends + replicate-weight survey: the analytical combination + # is rejected post-fit by ImputationDiDResults._aggregate_compute; + # replicate + n_bootstrap > 0 is rejected below, before any surface + # is materialized.) # Inert-config warning (no-silent-failures): the df_convention knob # moves only the pretrends lead regression's per-lead t/p/CI. Since - # the M-021 post-fit migration that inference is REACHABLE from any - # analytical pretrends=True fit via results.aggregate('event_study'), - # so the predicate is reachability-based, not aggregate-keyed: - # reachable iff pretrends AND not replicate-weight (the gate above - # rejects fit-time ES and the post-fit path fails closed too) AND - # (the deprecated fit-time ES/all was supplied OR n_bootstrap <= 0 — - # a bootstrapped fit builds no ES surface and post-fit aggregate() - # fails closed on it; validate_n_bootstrap rejects negatives at - # __init__, so 0 is the only reachable off value and `<= 0` is - # equivalent to `== 0` — kept as-is, no behavior change). Reachability-BASED, not exact: a - # fit whose bootstrap later FAILS (bootstrap_results=None) can still - # aggregate post-fit, so that corner warns spuriously — the warning - # fires before the bootstrap runs and cannot know. (The post-fit - # ``pretrend_test()`` reaches the lead helper too, but consumes only - # gamma/V_gamma — its joint Wald F denominator is knob-independent.) + # 4.0 that inference is REACHABLE from every pretrends=True fit via + # results.aggregate('event_study') - analytical fits recompute it + # from the kit, bootstrapped fits materialize the lead surface at + # fit and relay it - EXCEPT under a replicate-weight design, where + # the post-fit route fails closed (per-replicate lead refits are not + # implemented). So: reachable iff pretrends AND not replicate. + # (The post-fit ``pretrend_test()`` reaches the lead helper too, but + # consumes only gamma/V_gamma — its joint Wald F denominator is + # knob-independent.) _is_replicate_design = ( survey_design is not None and survey_design.replicate_method is not None ) - _lead_inference_reachable = ( - self.pretrends - and not _is_replicate_design - and (aggregate in ("event_study", "all") or self.n_bootstrap <= 0) - ) + _lead_inference_reachable = self.pretrends and not _is_replicate_design if self.df_convention != "residual" and not _lead_inference_reachable: warnings.warn( f"df_convention={self.df_convention!r} affects only the " "pretrends event-study per-lead inference (pretrends=True, " - "surfaced fit-time via the deprecated aggregate=" - "'event_study'/'all' or post-fit via " - "results.aggregate('event_study')); it has no effect on " - "this configuration.", + "surfaced via results.aggregate('event_study')); it has no " + "effect on this configuration.", UserWarning, stacklevel=2, ) @@ -764,56 +692,70 @@ def fit( overall_att, overall_se, alpha=self.alpha, df=_survey_df ) - # Event study and group aggregation (full-sample, for point estimates) - event_study_effects = None - group_effects = None - - if aggregate in ("event_study", "all"): - event_study_effects = self._aggregate_event_study( - df=df, - outcome=outcome, - unit=unit, - time=time, - first_treat=first_treat, - covariates=covariates, - omega_0_mask=omega_0_mask, - omega_1_mask=omega_1_mask, - unit_fe=unit_fe, - time_fe=time_fe, - grand_mean=grand_mean, - delta_hat=delta_hat, - cluster_var=cluster_var, - treatment_groups=treatment_groups, - balance_e=balance_e, - kept_cov_mask=kept_cov_mask, - survey_weights=survey_weights, - survey_df=_survey_df, - resolved_survey=(resolved_survey if not _uses_replicate_imp else None), - proj_cache=proj_cache, - ) - - if aggregate in ("group", "all"): - group_effects = self._aggregate_group( - df=df, - outcome=outcome, - unit=unit, - time=time, - first_treat=first_treat, - covariates=covariates, - omega_0_mask=omega_0_mask, - omega_1_mask=omega_1_mask, - unit_fe=unit_fe, - time_fe=time_fe, - grand_mean=grand_mean, - delta_hat=delta_hat, - cluster_var=cluster_var, - treatment_groups=treatment_groups, - kept_cov_mask=kept_cov_mask, - survey_weights=survey_weights, - survey_df=_survey_df, - resolved_survey=(resolved_survey if not _uses_replicate_imp else None), - proj_cache=proj_cache, - ) + # Event study and group aggregation (full-sample, for point estimates). + # Materialized at fit ONLY where the post-fit route cannot recompute + # it: a bootstrapped fit (n_bootstrap > 0) needs the ES + group + # targets inside the multiplier bootstrap (per-draw state is not + # retained), so the surfaces are built here with balance_e=None and + # RELAYED by results.aggregate('event_study') / ('group'). Analytic + # fits leave both None and recompute from the kit on demand. + # FAIL-SOFT by design: any exception while materializing leaves the + # surfaces None (the bootstrap then covers the overall ATT only - + # today's plain-fit path) and is reported once the bootstrap + # outcome is known (see the storage rule below). + event_study_effects: Optional[Dict[int, Dict[str, Any]]] = None + group_effects: Optional[Dict[Any, Dict[str, Any]]] = None + _surface_failure: Optional[BaseException] = None + + if self.n_bootstrap > 0: + try: + event_study_effects = self._aggregate_event_study( + df=df, + outcome=outcome, + unit=unit, + time=time, + first_treat=first_treat, + covariates=covariates, + omega_0_mask=omega_0_mask, + omega_1_mask=omega_1_mask, + unit_fe=unit_fe, + time_fe=time_fe, + grand_mean=grand_mean, + delta_hat=delta_hat, + cluster_var=cluster_var, + treatment_groups=treatment_groups, + balance_e=None, + kept_cov_mask=kept_cov_mask, + survey_weights=survey_weights, + survey_df=_survey_df, + resolved_survey=(resolved_survey if not _uses_replicate_imp else None), + proj_cache=proj_cache, + ) + group_effects = self._aggregate_group( + df=df, + outcome=outcome, + unit=unit, + time=time, + first_treat=first_treat, + covariates=covariates, + omega_0_mask=omega_0_mask, + omega_1_mask=omega_1_mask, + unit_fe=unit_fe, + time_fe=time_fe, + grand_mean=grand_mean, + delta_hat=delta_hat, + cluster_var=cluster_var, + treatment_groups=treatment_groups, + kept_cov_mask=kept_cov_mask, + survey_weights=survey_weights, + survey_df=_survey_df, + resolved_survey=(resolved_survey if not _uses_replicate_imp else None), + proj_cache=proj_cache, + ) + except Exception as e: # noqa: BLE001 - fail-soft by design + _surface_failure = e + event_study_effects = None + group_effects = None # Replicate variance: derive keys from actual outputs (after filtering) if _uses_replicate_imp: @@ -834,7 +776,7 @@ def fit( overall_att=overall_att, event_study_effects=event_study_effects, group_effects=group_effects, - balance_e=balance_e, + balance_e=None, survey_df_seed=_survey_df, ) overall_se = float(np.sqrt(max(_vcov_rep_imp[0, 0], 0.0))) @@ -883,15 +825,29 @@ def fit( "survey_weights": survey_weights, } - # Pre-compute cluster psi sums for bootstrap - psi_data = None + # Bootstrap stage: psi precompute + multiplier bootstrap. The + # engine draws the full weight matrix from the seeded RNG BEFORE any + # ES/group work, so the overall-ATT draws are the same with or + # without the surfaces. Attempt 1 runs WITH the materialized + # surfaces; if it fails while they are present (anything but an + # explicit NotImplementedError, which is re-raised), the surfaces + # are dropped and the stage is RETRIED ONCE overall-only - today's + # plain-fit path with the pre-4.0 failure semantics (psi failure -> + # warn + skip; _run_bootstrap failure -> propagates). On a seeded + # fit the retry reproduces attempt 1's draws exactly. + psi_data: Optional[Dict[str, Any]] = None + bootstrap_results: Optional[ImputationBootstrapResults] = None if self.n_bootstrap > 0 and n_valid > 0: - try: - # Extract survey weights for untreated obs (same as analytical path) - _sw_0 = survey_weights[omega_0_mask.values] if survey_weights is not None else None - # Extract survey weights for treated obs (event-study/group bootstrap paths) - _sw_1 = survey_weights[omega_1_mask.values] if survey_weights is not None else None - psi_data = self._precompute_bootstrap_psi( + # Extract survey weights for untreated obs (same as analytical path) + _sw_0 = survey_weights[omega_0_mask.values] if survey_weights is not None else None + # Extract survey weights for treated obs (event-study/group bootstrap paths) + _sw_1 = survey_weights[omega_1_mask.values] if survey_weights is not None else None + + def _precompute_psi( + _es: Optional[Dict[int, Dict[str, Any]]], + _grp: Optional[Dict[Any, Dict[str, Any]]], + ) -> Dict[str, Any]: + return self._precompute_bootstrap_psi( df=df, outcome=outcome, unit=unit, @@ -907,34 +863,79 @@ def fit( cluster_var=cluster_var, kept_cov_mask=kept_cov_mask, overall_weights=overall_weights, - event_study_effects=event_study_effects, - group_effects=group_effects, + event_study_effects=_es, + group_effects=_grp, treatment_groups=treatment_groups, tau_hat=tau_hat, - balance_e=balance_e, + balance_e=None, survey_weights_0=_sw_0, survey_weights_1=_sw_1, proj_cache=proj_cache, ) - except Exception as e: - warnings.warn( - f"Bootstrap pre-computation failed: {e}. " "Skipping bootstrap inference.", - UserWarning, - stacklevel=2, - ) - psi_data = None - - # Bootstrap - bootstrap_results = None - if self.n_bootstrap > 0 and psi_data is not None: - bootstrap_results = self._run_bootstrap( - original_att=overall_att, - original_event_study=event_study_effects, - original_group=group_effects, - psi_data=psi_data, - resolved_survey=resolved_survey, + + _surfaces_present = event_study_effects is not None or group_effects is not None + if _surfaces_present: + try: + psi_data = _precompute_psi(event_study_effects, group_effects) + bootstrap_results = self._run_bootstrap( + original_att=overall_att, + original_event_study=event_study_effects, + original_group=group_effects, + psi_data=psi_data, + resolved_survey=resolved_survey, + ) + except NotImplementedError: + raise # explicit rejections are never retried around + except Exception as e: # noqa: BLE001 - fail-soft by design + _surface_failure = e + event_study_effects = None + group_effects = None + psi_data = None + bootstrap_results = None + if not _surfaces_present or bootstrap_results is None: + # Plain overall-only stage (no surfaces), or the one-shot + # retry after a failure with surfaces - identical semantics. + try: + psi_data = _precompute_psi(None, None) + except Exception as e: + warnings.warn( + f"Bootstrap pre-computation failed: {e}. " "Skipping bootstrap inference.", + UserWarning, + stacklevel=2, + ) + psi_data = None + if psi_data is not None: + bootstrap_results = self._run_bootstrap( + original_att=overall_att, + original_event_study=None, + original_group=None, + psi_data=psi_data, + resolved_survey=resolved_survey, + ) + + # Storage rule: the legacy fields hold a surface IFF the fit carries + # bootstrap inference. Without it (n_valid == 0 skipped the stage, or + # the bootstrap failed outright and the fit fell back to analytical + # inference) an analytical surface must never sit on a bootstrapped + # fit where summary()/to_dataframe()/the DiagnosticReport would read + # it as the fit's inference; aggregate() recomputes analytically + # there, exactly as on an n_bootstrap=0 fit. + if self.n_bootstrap > 0 and bootstrap_results is None: + event_study_effects = None + group_effects = None + elif bootstrap_results is not None and _surface_failure is not None: + warnings.warn( + f"{type(self).__name__}: the event-study/group surfaces could not " + f"be materialized on this bootstrapped fit ({_surface_failure}); " + "the bootstrap covered the overall ATT only, so " + "aggregate('event_study') / aggregate('group') are unavailable " + "here - re-fit with n_bootstrap=0 for the analytical post-fit " + "recompute.", + UserWarning, + stacklevel=2, ) + if bootstrap_results is not None: # Update inference with bootstrap results overall_se = bootstrap_results.overall_att_se overall_t = ( @@ -1046,9 +1047,9 @@ def fit( ) # Attach the post-fit aggregation kit (M-021/M-118). Unconditional — - # including bootstrap fits, whose gate lives in _aggregate_compute + # including bootstrap fits, whose relay lives in _aggregate_compute # (a FAILED bootstrap leaves bootstrap_results=None and the fit - # aggregates normally). + # aggregates normally from the kit). self.results_._aggregation_kit = _build_imputation_aggregation_kit( fit_data=self._fit_data, treatment_groups=treatment_groups, @@ -1405,5 +1406,5 @@ def _build_imputation_aggregation_kit( alpha=alpha, anticipation=anticipation, cband=False, # no simultaneous-band concept on this estimator - bootstrap=None, # replay not wired; recompute levels fail closed ('simple' relays, M-027) + bootstrap=None, # no replay: bootstrapped fits RELAY the fit-time surfaces (4.0 PR 3c) ) diff --git a/diff_diff/imputation_aggregation.py b/diff_diff/imputation_aggregation.py index a9feaebc..37927677 100644 --- a/diff_diff/imputation_aggregation.py +++ b/diff_diff/imputation_aggregation.py @@ -1723,10 +1723,11 @@ def _replicate_override_aggregates( dicts are non-None. ``compute_replicate_refit_variance`` validates replicates JOINTLY (all-finite rows, ``survey.py``), so inference is LEVEL-MATCHED: post-fit ``aggregate('event_study')`` replays - ``[overall, ES]`` and reproduces ``fit(aggregate='event_study')`` - exactly; a ``fit(aggregate='all')`` surface is NOT the equivalence - target when a replicate NaNs on exactly one family's targets (its - joint stack drops that replicate for every row). + ``[overall, ES]`` and ``aggregate('group')`` replays + ``[overall, groups]``; a joint ``[overall, ES, groups]`` stack is + NOT the equivalence target when a replicate NaNs on exactly one + family's targets (its joint stack drops that replicate for every + row). Overrides ``se``/``t_stat``/``p_value``/``conf_int`` (and, on group rows, the ``df_used`` provenance key) IN PLACE on the passed dicts diff --git a/diff_diff/imputation_results.py b/diff_diff/imputation_results.py index f9e31587..84c979ba 100644 --- a/diff_diff/imputation_results.py +++ b/diff_diff/imputation_results.py @@ -12,7 +12,12 @@ import numpy as np import pandas as pd -from diff_diff.aggregation import AggregationMixin, AggregationResult, build_total_relay_row +from diff_diff.aggregation import ( + AggregationMixin, + AggregationResult, + absent_bootstrap_surface_message, + build_total_relay_row, +) from diff_diff.imputation_aggregation import _ImputationAggregationMixin from diff_diff.results import _format_survey_block, _get_significance_stars from diff_diff.results_base import ( @@ -261,37 +266,59 @@ def _aggregate_compute( # Per-level bootstrap policy (v4-design section 6, converged with row # M-027): 'simple' is a bit-exact RELAY of the stored overall quintet # - faithful under any inference regime, bootstrap included - so it - # dispatches BEFORE the bootstrap gate. Only the RECOMPUTE levels - # below fail closed on bootstrapped fits. (This supersedes the + # dispatches BEFORE the bootstrap branch. (This supersedes the # uniform-conservatism decision recorded with M-021; its rationale is # honored by the relay's NaN df column.) if level == "simple": return self._aggregate_simple_result(kit) if level == "total": return self._aggregate_total_result(kit) - if self.bootstrap_results is not None: - raise NotImplementedError( - f"aggregate({level!r}) is not yet available on a bootstrapped " - "fit (n_bootstrap > 0): the per-target bootstrap draws are " - "not retained, so post-fit re-aggregation cannot replay " - "percentile inference and analytical inference would " - "misrepresent the fit. aggregate('simple') and, where " - "supported, aggregate('total') relay the stored " - "bootstrap inference and remain available; otherwise re-fit " - "with the aggregation you need, or use n_bootstrap=0." - ) bk = dict(kit.bookkeeping) + if self.bootstrap_results is not None: # the fit's bootstrap SUCCEEDED + # 4.0 PR 3c: bootstrapped fits materialize ES + group at fit + # (balance_e=None, percentile overrides applied, df cleared); no + # replay state is retained, so RELAY the stored surface with + # kit-snapshot provenance (StackedDiD view mechanics). The relay + # reads the MUTABLE public field by definition (a view); only + # alpha/anticipation/survey metadata come from kit snapshots. + cls_name = type(self).__name__ + if level == "group": + if self.group_effects is None: + raise ValueError(absent_bootstrap_surface_message(cls_name, "group")) + return self._group_effects_to_aggregation(self.group_effects, kit) + if balance_e is not None: + raise NotImplementedError( + "aggregate('event_study', balance_e=...) is not available on a " + "bootstrapped ImputationDiD fit: the fit-time percentile surface " + "was computed with balance_e=None and the per-target draws are " + "not retained. Call aggregate('event_study') for the stored " + "surface, or re-fit with n_bootstrap=0 to recompute with " + "balance_e." + ) + if self.event_study_effects is None: + raise ValueError(absent_bootstrap_surface_message(cls_name, "event_study")) + meta = bk["survey_metadata"] + carrier = dataclasses.replace( + self, + survey_metadata=(dataclasses.replace(meta) if meta is not None else None), + anticipation=kit.anticipation, + alpha=kit.alpha, + event_study_df=None, # percentile inference governs no df + ) + return build_event_study_surface(carrier) + # bootstrap_results is None: n_bootstrap == 0, OR the fit's bootstrap + # failed and the fit fell back to analytical inference (imputation.py + # stores no surface then) - the analytical kit RECOMPUTE below matches + # the fit's analytical provenance. if level == "event_study" and bk["uses_replicate"] and bk["pretrends"]: - # The same unsupported combination fit(aggregate='event_study') - # rejects: the pre-period lead regression's per-replicate refits - # are not implemented, and the replicate replay re-estimates + # The pre-period lead regression's per-replicate refits are not + # implemented, and the replicate replay re-estimates # post-treatment targets only. raise NotImplementedError( "aggregate('event_study') is not available on this fit: it " "used pretrends=True with a replicate-weight survey design, " "and the pre-period lead regression's per-replicate refits " - "are not yet implemented (fit(aggregate='event_study') " - "rejects the same combination). Re-fit with pretrends=False, " + "are not yet implemented. Re-fit with pretrends=False, " "or use an analytical (strata/PSU/FPC) survey design." ) # Fresh throwaway host per call, populated from KIT snapshots only @@ -333,8 +360,8 @@ def _aggregate_compute( if level == "group": effects = agg._aggregate_group(**common) if bk["uses_replicate"]: - # LEVEL-MATCHED replay: [overall, groups] - reproduces - # fit(aggregate='group') exactly (see the replay docstring). + # LEVEL-MATCHED replay: [overall, groups] (see the replay + # docstring). agg._replicate_override_aggregates( df=bk["df"], outcome=bk["outcome"], @@ -792,12 +819,11 @@ def to_dataframe(self, level: str = "observation") -> pd.DataFrame: elif level == "event_study": if self.event_study_effects is None: raise ValueError( - "Event study effects not computed. Aggregate post-fit " - "instead - results.aggregate('event_study') returns the " - "EventStudyResults container (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or use the deprecated " - "fit-time aggregate=; a result unpickled from a pre-3.9 " - "release carries no kit and must be re-fit)." + "Event study effects not stored on this fit. Use " + "results.aggregate('event_study') - analytical fits " + "recompute from the retained kit; bootstrapped fits relay " + "the surface computed at fit. A result unpickled from a " + "pre-3.9 release carries no kit and must be re-fit." ) rows = [] for h, data in sorted(self.event_study_effects.items()): @@ -818,12 +844,11 @@ def to_dataframe(self, level: str = "observation") -> pd.DataFrame: elif level == "group": if self.group_effects is None: raise ValueError( - "Group effects not computed. Aggregate post-fit instead " - "- results.aggregate('group') returns the " - "AggregationResult container (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or use the deprecated " - "fit-time aggregate=; a result unpickled from a pre-3.9 " - "release carries no kit and must be re-fit)." + "Group effects not stored on this fit. Use " + "results.aggregate('group') - analytical fits recompute " + "from the retained kit; bootstrapped fits relay the " + "surface computed at fit. A result unpickled from a " + "pre-3.9 release carries no kit and must be re-fit." ) rows = [] for g, data in sorted(self.group_effects.items()): diff --git a/diff_diff/practitioner.py b/diff_diff/practitioner.py index e34d8e3d..48326c45 100644 --- a/diff_diff/practitioner.py +++ b/diff_diff/practitioner.py @@ -650,22 +650,16 @@ def _handle_imputation(results: Any): "ImputationDiD aggregates post-fit from its panel-backed kit " "(M-021) - no refit needed." if getattr(results, "bootstrap_results", None) is None - else "This fit is BOOTSTRAPPED: the post-fit event-study/group " - "recompute levels raise on bootstrap fits, while " - "aggregate('simple') and, where supported, " - "aggregate('total') relay the stored inference - " - "refit with the deprecated fit-time aggregation (or " - "n_bootstrap=0) to obtain the recomputed surfaces." + else "This fit is BOOTSTRAPPED: ImputationDiD aggregates post-fit " + "(M-021) - no refit needed; bootstrapped fits relay the " + "percentile event-study/group surfaces computed at fit " + "(balance_e unavailable there - re-fit with n_bootstrap=0 " + "for a balanced window)." ), code=( "# Aggregate post-fit - no refit needed:\n" "print(results.aggregate('group').to_dataframe()) # Per-cohort ATTs\n" "print(results.aggregate('event_study').to_dataframe()) # Dynamic effects" - if getattr(results, "bootstrap_results", None) is None - else "# Bootstrap fit: aggregate at fit time (deprecated kwarg):\n" - "results = imp.fit(data, ..., aggregate='all')\n" - "print(results.group_effects) # Per-cohort ATTs\n" - "print(results.event_study_effects) # Dynamic effects" ), priority="medium", # NON-STEPS key (the M-024 "sub_experiment_balance" lesson): @@ -709,22 +703,16 @@ def _handle_two_stage(results: Any): "TwoStageDiD aggregates post-fit from its panel-backed kit " "(M-022) - no refit needed." if getattr(results, "bootstrap_results", None) is None - else "This fit is BOOTSTRAPPED: the post-fit event-study/group " - "recompute levels raise on bootstrap fits, while " - "aggregate('simple') and, where supported, " - "aggregate('total') relay the stored inference - " - "refit with the deprecated fit-time aggregation (or " - "n_bootstrap=0) to obtain the recomputed surfaces." + else "This fit is BOOTSTRAPPED: TwoStageDiD aggregates post-fit " + "(M-022) - no refit needed; bootstrapped fits relay the " + "percentile event-study/group surfaces computed at fit " + "(balance_e unavailable there - re-fit with n_bootstrap=0 " + "for a balanced window)." ), code=( "# Aggregate post-fit - no refit needed:\n" "print(results.aggregate('group').to_dataframe()) # Per-cohort ATTs\n" "print(results.aggregate('event_study').to_dataframe()) # Dynamic effects" - if getattr(results, "bootstrap_results", None) is None - else "# Bootstrap fit: aggregate at fit time (deprecated kwarg):\n" - "results = ts.fit(data, ..., aggregate='all')\n" - "print(results.group_effects) # Per-cohort ATTs\n" - "print(results.event_study_effects) # Dynamic effects" ), priority="medium", # NON-STEPS key (the M-024 "sub_experiment_balance" lesson): @@ -1232,18 +1220,16 @@ def _handle_continuous(results: Any): baker_step=7, label="Aggregate post-fit (event study / dose / simple)", why=( - "The fit-time aggregate= kwarg is deprecated (row M-025; " - "removed in 4.0). Aggregate as a post-fit step instead: " + "Aggregate as a post-fit step (row M-025): " "aggregate('event_study') recomputes the binarized event " - "study from the retained kit (analytical fits only - on a " - "bootstrapped fit it raises; re-fit with n_bootstrap=0 or " - "use the deprecated fit-time aggregate='eventstudy' until " - "4.0), while aggregate('dose') and aggregate('simple') are " + "study from the retained kit on analytic fits and relays " + "the percentile surface computed at fit on bootstrapped " + "fits, while aggregate('dose') and aggregate('simple') are " "views over the always-computed curves and overall " "ATT/ACRT, available on any fit." ), code=( - "es = results.aggregate('event_study') # analytical fits\n" + "es = results.aggregate('event_study')\n" "dose_table = results.aggregate('dose')\n" "overall = results.aggregate('simple') # att + acrt rows" ), diff --git a/diff_diff/results_base.py b/diff_diff/results_base.py index e1634e95..5f524393 100644 --- a/diff_diff/results_base.py +++ b/diff_diff/results_base.py @@ -774,12 +774,14 @@ def summary(self, alpha: Optional[float] = None) -> str: # Post-fit only by design (no fit-time surface ever exists). "DMLDiDResults": "call results.aggregate('event_study')", "ImputationDiDResults": ( - "call results.aggregate('event_study') (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or the deprecated fit-time aggregate=)" + "call results.aggregate('event_study') (a bootstrapped fit from before " + "4.0 stores no surface - re-fit with diff-diff >= 4.0, which computes " + "it at fit)" ), "TwoStageDiDResults": ( - "call results.aggregate('event_study') (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or the deprecated fit-time aggregate=)" + "call results.aggregate('event_study') (a bootstrapped fit from before " + "4.0 stores no surface - re-fit with diff-diff >= 4.0, which computes " + "it at fit)" ), # Absence only possible on pre-3.9 pickles: 3.9+ fits always # materialize the surface (row M-024). @@ -787,8 +789,9 @@ def summary(self, alpha: Optional[float] = None) -> str: "StaggeredTripleDiffResults": "refit with aggregate='event_study' (or 'all')", "EfficientDiDResults": "call results.aggregate('event_study')", "ContinuousDiDResults": ( - "call results.aggregate('event_study') (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or the deprecated fit-time aggregate=)" + "call results.aggregate('event_study') (a bootstrapped fit from before " + "4.0 stores no surface - re-fit with diff-diff >= 4.0, which computes " + "it at fit)" ), "WooldridgeDiDResults": "call results.aggregate('event_study') first", "SpilloverDiDResults": "refit with event_study=True", diff --git a/diff_diff/two_stage.py b/diff_diff/two_stage.py index d38a7328..19f1fa56 100644 --- a/diff_diff/two_stage.py +++ b/diff_diff/two_stage.py @@ -31,7 +31,6 @@ from scipy.sparse.linalg import factorized as sparse_factorized from diff_diff._base import BaseEstimator -from diff_diff._deprecation import NOT_SUPPLIED from diff_diff.aggregation import AggregationKit from diff_diff.conley import ( ConleyMetric, @@ -1363,8 +1362,6 @@ def fit( time: str, first_treat: str, covariates: Optional[List[str]] = None, - aggregate: Any = NOT_SUPPLIED, - balance_e: Any = NOT_SUPPLIED, survey_design: Optional["SurveyDesign"] = None, ) -> TwoStageDiDResults: """ @@ -1385,20 +1382,6 @@ def fit( Use 0 (or np.inf) for never-treated units. covariates : list of str, optional List of covariate column names. - aggregate : str, optional - DEPRECATED (3.9, removed in 4.0; row M-022): aggregate as a - post-fit step instead — ``results.aggregate('event_study')`` / - ``.aggregate('group')`` / ``.aggregate('simple')`` / - ``.aggregate('total')``. Supplying - ANY value (``None`` included) warns; the deprecated path still - works and returns exactly the numbers it always did - (fit-time mode: None/"simple" overall only, "event_study", - "group", or "all"). - balance_e : int, optional - DEPRECATED (3.9, removed in 4.0; row M-119): moves onto - ``results.aggregate('event_study', balance_e=...)``. Restricts - the event study to cohorts observed at every relative time in - ``[-balance_e, max_h]`` (the balanced-window rule). survey_design : SurveyDesign, optional Survey design specification for design-based inference. Supports pweight only (aweight/fweight raise ValueError). Supports strata, @@ -1416,35 +1399,6 @@ def fit( ValueError If required columns are missing or data validation fails. """ - # M-022/M-119 deprecation shim (CS-style joint warning): a plain - # fit() never warns; supplying EITHER param with ANY value (None - # included) warns once, then the legacy routing below runs - # unchanged - the deprecated path returns exactly the numbers it - # always did (no new value validation; unknown strings still act - # like None). The post-fit successor validates its own vocabulary. - _deprecated_passed = [ - n - for n, v in (("aggregate", aggregate), ("balance_e", balance_e)) - if v is not NOT_SUPPLIED - ] - if _deprecated_passed: - _args = " / ".join(f"{n}=" for n in _deprecated_passed) - warnings.warn( - f"TwoStageDiD.fit({_args}) is deprecated and will be " - "removed in 4.0. Fit once, then aggregate as a post-fit " - "step: results = TwoStageDiD().fit(...); " - "results.aggregate('event_study') / .aggregate('group') / " - ".aggregate('simple') / .aggregate('total'). balance_e moves onto aggregate() " - "alongside it: results.aggregate('event_study', " - "balance_e=2).", - FutureWarning, - stacklevel=2, - ) - if aggregate is NOT_SUPPLIED: - aggregate = None - if balance_e is NOT_SUPPLIED: - balance_e = None - # Re-validate vcov_type at fit-time: set_params validates eagerly # (BaseEstimator probe re-init), so this only catches DIRECT # attribute mutation (est.vcov_type = ...). @@ -1796,61 +1750,77 @@ def fit( overall_att, overall_se, alpha=self.alpha, df=_survey_df ) - # Event study and group aggregation (full-sample, for point estimates) - event_study_effects = None - group_effects = None + # Event study and group aggregation (full-sample, for point estimates). + # Materialized at fit ONLY where the post-fit route cannot recompute + # it: a bootstrapped fit (n_bootstrap > 0) needs the ES + group + # targets inside the multiplier bootstrap (per-draw state is not + # retained), so the surfaces are built here with balance_e=None and + # RELAYED by results.aggregate('event_study') / ('group'). Analytic + # fits leave both None and recompute from the kit on demand. + # FAIL-SOFT by design: any exception while materializing leaves the + # surfaces None (the bootstrap then covers the overall ATT only - + # today's plain-fit path) and is reported once the bootstrap + # outcome is known (see the storage rule below). + event_study_effects: Optional[Dict[int, Dict[str, Any]]] = None + group_effects: Optional[Dict[Any, Dict[str, Any]]] = None _es_vcov_ts: Optional[np.ndarray] = None _es_vcov_index_ts: Optional[List[int]] = None + _surface_failure: Optional[BaseException] = None - if aggregate in ("event_study", "all"): - event_study_effects, _es_vcov_ts, _es_vcov_index_ts = self._stage2_event_study( - df=df, - unit=unit, - time=time, - first_treat=first_treat, - covariates=covariates, - omega_0_mask=omega_0_mask, - omega_1_mask=omega_1_mask, - unit_fe=unit_fe, - time_fe=time_fe, - grand_mean=grand_mean, - delta_hat=delta_hat, - cluster_var=cluster_var, - treatment_groups=treatment_groups, - ref_period=ref_period, - balance_e=balance_e, - kept_cov_mask=kept_cov_mask, - survey_weights=survey_weights, - survey_weight_type=survey_weight_type, - survey_df=_survey_df, - resolved_survey=(resolved_survey if not _uses_replicate_ts else None), - score_pad_mask=score_pad_mask_arg, - cluster_ids_full=cluster_ids_full_arg, - ) - - if aggregate in ("group", "all"): - group_effects = self._stage2_group( - df=df, - unit=unit, - time=time, - first_treat=first_treat, - covariates=covariates, - omega_0_mask=omega_0_mask, - omega_1_mask=omega_1_mask, - unit_fe=unit_fe, - time_fe=time_fe, - grand_mean=grand_mean, - delta_hat=delta_hat, - cluster_var=cluster_var, - treatment_groups=treatment_groups, - kept_cov_mask=kept_cov_mask, - survey_weights=survey_weights, - survey_weight_type=survey_weight_type, - survey_df=_survey_df, - resolved_survey=(resolved_survey if not _uses_replicate_ts else None), - score_pad_mask=score_pad_mask_arg, - cluster_ids_full=cluster_ids_full_arg, - ) + if self.n_bootstrap > 0: + try: + event_study_effects, _es_vcov_ts, _es_vcov_index_ts = self._stage2_event_study( + df=df, + unit=unit, + time=time, + first_treat=first_treat, + covariates=covariates, + omega_0_mask=omega_0_mask, + omega_1_mask=omega_1_mask, + unit_fe=unit_fe, + time_fe=time_fe, + grand_mean=grand_mean, + delta_hat=delta_hat, + cluster_var=cluster_var, + treatment_groups=treatment_groups, + ref_period=ref_period, + balance_e=None, + kept_cov_mask=kept_cov_mask, + survey_weights=survey_weights, + survey_weight_type=survey_weight_type, + survey_df=_survey_df, + resolved_survey=(resolved_survey if not _uses_replicate_ts else None), + score_pad_mask=score_pad_mask_arg, + cluster_ids_full=cluster_ids_full_arg, + ) + group_effects = self._stage2_group( + df=df, + unit=unit, + time=time, + first_treat=first_treat, + covariates=covariates, + omega_0_mask=omega_0_mask, + omega_1_mask=omega_1_mask, + unit_fe=unit_fe, + time_fe=time_fe, + grand_mean=grand_mean, + delta_hat=delta_hat, + cluster_var=cluster_var, + treatment_groups=treatment_groups, + kept_cov_mask=kept_cov_mask, + survey_weights=survey_weights, + survey_weight_type=survey_weight_type, + survey_df=_survey_df, + resolved_survey=(resolved_survey if not _uses_replicate_ts else None), + score_pad_mask=score_pad_mask_arg, + cluster_ids_full=cluster_ids_full_arg, + ) + except Exception as e: # noqa: BLE001 - fail-soft by design + _surface_failure = e + event_study_effects = None + group_effects = None + _es_vcov_ts = None + _es_vcov_index_ts = None # Replicate variance override: derive keys from actual outputs, then refit _n_valid_rep_ts = None @@ -1872,7 +1842,7 @@ def fit( cluster_var=cluster_var, treatment_groups=treatment_groups, ref_period=ref_period, - balance_e=balance_e, + balance_e=None, keep_mask=keep_mask, resolved_survey=resolved_survey, overall_att=overall_att, @@ -1905,10 +1875,26 @@ def fit( treated_df["weight"] = 0.0 # ---- Bootstrap ---- - bootstrap_results = None + # The engine draws the full weight matrix from the seeded RNG BEFORE + # any ES/group work, so the overall-ATT draws are the same with or + # without the surfaces. Attempt 1 runs WITH the materialized + # surfaces (aggregate="all" is the private selector gating the + # ES/group draws); it counts as failed-with-surfaces if it RAISES + # (anything but NotImplementedError, re-raised as today) or RETURNS + # None (an LSMR-unconverged ES/group column) while surfaces are + # present. Then the surfaces are dropped and the bootstrap is + # RETRIED ONCE overall-only - the pre-4.0 plain-fit path with its + # catch-all semantics (failure -> "Bootstrap failed" warning + + # analytical overall). On a seeded fit the retry reproduces attempt + # 1's draws exactly. + bootstrap_results: Optional[TwoStageBootstrapResults] = None if self.n_bootstrap > 0: - try: - bootstrap_results = self._run_bootstrap( + + def _run_stage( + _es: Optional[Dict[int, Dict[str, Any]]], + _grp: Optional[Dict[Any, Dict[str, Any]]], + ) -> Optional[TwoStageBootstrapResults]: + return self._run_bootstrap( df=df, unit=unit, time=time, @@ -1924,18 +1910,67 @@ def fit( kept_cov_mask=kept_cov_mask, treatment_groups=treatment_groups, ref_period=ref_period, - balance_e=balance_e, + balance_e=None, original_att=overall_att, - original_event_study=event_study_effects, - original_group=group_effects, - aggregate=aggregate, + original_event_study=_es, + original_group=_grp, + aggregate=("all" if (_es is not None or _grp is not None) else None), resolved_survey=resolved_survey, ) - except NotImplementedError: - raise # Don't swallow explicit rejections (e.g. lonely_psu="adjust") - except Exception as e: + + _surfaces_present = event_study_effects is not None or group_effects is not None + if _surfaces_present: + try: + bootstrap_results = _run_stage(event_study_effects, group_effects) + if bootstrap_results is None: + raise RuntimeError( + "the multiplier bootstrap returned no results for the " + "event-study/group targets" + ) + except NotImplementedError: + raise # Don't swallow explicit rejections (e.g. lonely_psu="adjust") + except Exception as e: # noqa: BLE001 - fail-soft by design + _surface_failure = e + event_study_effects = None + group_effects = None + _es_vcov_ts = None + _es_vcov_index_ts = None + bootstrap_results = None + if not _surfaces_present or bootstrap_results is None: + # Plain overall-only stage (no surfaces), or the one-shot + # retry after a failure with surfaces - identical semantics. + try: + bootstrap_results = _run_stage(None, None) + except NotImplementedError: + raise # Don't swallow explicit rejections (e.g. lonely_psu="adjust") + except Exception as e: + warnings.warn( + f"Bootstrap failed: {e}. Skipping bootstrap inference.", + UserWarning, + stacklevel=2, + ) + bootstrap_results = None + + # Storage rule: the legacy fields hold a surface IFF the fit + # carries bootstrap inference. A failed bootstrap falls back to + # analytical inference, and an analytical surface must never sit + # on a bootstrapped fit where summary()/to_dataframe()/the + # DiagnosticReport would read it as the fit's inference; + # aggregate() recomputes analytically there, exactly as on an + # n_bootstrap=0 fit. + if bootstrap_results is None: + event_study_effects = None + group_effects = None + _es_vcov_ts = None + _es_vcov_index_ts = None + elif _surface_failure is not None: warnings.warn( - f"Bootstrap failed: {e}. Skipping bootstrap inference.", + f"{type(self).__name__}: the event-study/group surfaces could not " + f"be materialized on this bootstrapped fit ({_surface_failure}); " + "the bootstrap covered the overall ATT only, so " + "aggregate('event_study') / aggregate('group') are unavailable " + "here - re-fit with n_bootstrap=0 for the analytical post-fit " + "recompute.", UserWarning, stacklevel=2, ) @@ -2075,9 +2110,9 @@ def fit( ) # Attach the post-fit aggregation kit (M-022/M-119). Unconditional — - # including bootstrap fits, whose gate lives in _aggregate_compute + # including bootstrap fits, whose relay lives in _aggregate_compute # (a FAILED bootstrap leaves bootstrap_results=None and the fit - # aggregates normally). + # aggregates normally from the kit). self.results_._aggregation_kit = _build_twostage_aggregation_kit( df=df, outcome=outcome, @@ -2329,5 +2364,5 @@ def _build_twostage_aggregation_kit( alpha=alpha, anticipation=anticipation, cband=False, # no simultaneous-band concept on this estimator - bootstrap=None, # replay not wired; recompute levels fail closed ('simple' relays, M-027) + bootstrap=None, # no replay: bootstrapped fits RELAY the fit-time surfaces (4.0 PR 3c) ) diff --git a/diff_diff/two_stage_aggregation.py b/diff_diff/two_stage_aggregation.py index 37692131..603fa48e 100644 --- a/diff_diff/two_stage_aggregation.py +++ b/diff_diff/two_stage_aggregation.py @@ -1384,8 +1384,8 @@ def _replay_replicate_inference( dicts are non-None. ``compute_replicate_refit_variance`` validates replicates JOINTLY (all-finite rows), so inference is LEVEL-MATCHED: post-fit ``aggregate('event_study')`` replays ``[overall, ES]`` and - reproduces ``fit(aggregate='event_study')`` exactly; a - ``fit(aggregate='all')`` surface is NOT the equivalence target when + ``aggregate('group')`` replays ``[overall, groups]``; a joint + ``[overall, ES, groups]`` stack is NOT the equivalence target when a replicate NaNs on exactly one family's targets. Overrides ``se``/``t_stat``/``p_value``/``conf_int`` IN PLACE on the diff --git a/diff_diff/two_stage_results.py b/diff_diff/two_stage_results.py index 19e56105..709cd455 100644 --- a/diff_diff/two_stage_results.py +++ b/diff_diff/two_stage_results.py @@ -12,7 +12,12 @@ import numpy as np import pandas as pd -from diff_diff.aggregation import AggregationMixin, AggregationResult, build_total_relay_row +from diff_diff.aggregation import ( + AggregationMixin, + AggregationResult, + absent_bootstrap_surface_message, + build_total_relay_row, +) from diff_diff.results import _format_survey_block, _get_significance_stars from diff_diff.results_base import ( BaseResults, @@ -250,26 +255,54 @@ def _aggregate_compute( # Per-level bootstrap policy (v4-design section 6, converged with row # M-027): 'simple' is a bit-exact RELAY of the stored overall quintet # - faithful under any inference regime, bootstrap included - so it - # dispatches BEFORE the bootstrap gate. Only the RECOMPUTE levels - # below fail closed on bootstrapped fits. (This supersedes the + # dispatches BEFORE the bootstrap branch. (This supersedes the # uniform-conservatism decision recorded with M-022; its rationale is # honored by the relay's NaN df column.) if level == "simple": return self._aggregate_simple_result(kit) if level == "total": return self._aggregate_total_result(kit) - if self.bootstrap_results is not None: - raise NotImplementedError( - f"aggregate({level!r}) is not yet available on a bootstrapped " - "fit (n_bootstrap > 0): the per-target bootstrap draws are " - "not retained, so post-fit re-aggregation cannot replay " - "percentile inference and analytical inference would " - "misrepresent the fit. aggregate('simple') and, where " - "supported, aggregate('total') relay the stored " - "bootstrap inference and remain available; otherwise re-fit " - "with the aggregation you need, or use n_bootstrap=0." - ) bk = dict(kit.bookkeeping) + if self.bootstrap_results is not None: # the fit's bootstrap SUCCEEDED + # 4.0 PR 3c: bootstrapped fits materialize ES + group at fit + # (balance_e=None, percentile overrides applied, vcov/df cleared); + # no replay state is retained, so RELAY the stored surface with + # kit-snapshot provenance (StackedDiD view mechanics). The relay + # reads the MUTABLE public field by definition (a view); only + # alpha/anticipation/survey metadata come from kit snapshots. + cls_name = type(self).__name__ + if level == "group": + if self.group_effects is None: + raise ValueError(absent_bootstrap_surface_message(cls_name, "group")) + # The bootstrap override replaced se/p/CI - no analytical df + # may leak beside percentile inference -> NaN df column. + return self._group_effects_to_aggregation(self.group_effects, kit, group_df=None) + if balance_e is not None: + raise NotImplementedError( + "aggregate('event_study', balance_e=...) is not available on a " + "bootstrapped TwoStageDiD fit: the fit-time percentile surface " + "was computed with balance_e=None and the per-target draws are " + "not retained. Call aggregate('event_study') for the stored " + "surface, or re-fit with n_bootstrap=0 to recompute with " + "balance_e." + ) + if self.event_study_effects is None: + raise ValueError(absent_bootstrap_surface_message(cls_name, "event_study")) + meta = bk["survey_metadata"] + carrier = dataclasses.replace( + self, + survey_metadata=(dataclasses.replace(meta) if meta is not None else None), + anticipation=kit.anticipation, + alpha=kit.alpha, + event_study_vcov=None, + event_study_vcov_index=None, + event_study_df=None, # percentile inference governs no df + ) + return build_event_study_surface(carrier) + # bootstrap_results is None: n_bootstrap == 0, OR the fit's bootstrap + # failed and the fit fell back to analytical inference (two_stage.py + # stores no surface then) - the analytical kit RECOMPUTE below matches + # the fit's analytical provenance. # Fresh throwaway host per call, populated from KIT snapshots only # (estimator/config mutation after fit() must not leak in). agg = _TwoStageKitAggregator( @@ -304,8 +337,8 @@ def _aggregate_compute( effects = agg._stage2_group(**common) replay_df_g: Optional[int] = bk["survey_df_stage2"] if bk["uses_replicate"]: - # LEVEL-MATCHED replay: [overall, groups] - reproduces - # fit(aggregate='group') exactly (see the replay docstring). + # LEVEL-MATCHED replay: [overall, groups] (see the replay + # docstring). _, _, replay_df_g = agg._replay_replicate_inference( df=bk["df"], outcome=bk["outcome"], @@ -356,7 +389,7 @@ def _aggregate_compute( # Carrier + shared builder, reproducing fit's M-092 mode gates # exactly (two_stage.py): analytical -> recomputed V + index + the # finite-and->0 df scalar; replicate -> vcov/index None with the - # REPLAYED level-matched df; bootstrap unreachable (failed closed + # REPLAYED level-matched df; bootstrap unreachable (relayed # above). The carrier's metadata is a copy-on-use of the KIT's # fit-final metadata copy (never the mutable public field). if bk["uses_replicate"]: @@ -752,12 +785,11 @@ def to_dataframe(self, level: str = "event_study") -> pd.DataFrame: elif level == "event_study": if self.event_study_effects is None: raise ValueError( - "Event study effects not computed. Aggregate post-fit " - "instead - results.aggregate('event_study') returns the " - "EventStudyResults container (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or use the deprecated " - "fit-time aggregate=; a result unpickled from a pre-3.9 " - "release carries no kit and must be re-fit)." + "Event study effects not stored on this fit. Use " + "results.aggregate('event_study') - analytical fits " + "recompute from the retained kit; bootstrapped fits relay " + "the surface computed at fit. A result unpickled from a " + "pre-3.9 release carries no kit and must be re-fit." ) rows = [] for h, data in sorted(self.event_study_effects.items()): @@ -778,12 +810,11 @@ def to_dataframe(self, level: str = "event_study") -> pd.DataFrame: elif level == "group": if self.group_effects is None: raise ValueError( - "Group effects not computed. Aggregate post-fit instead " - "- results.aggregate('group') returns the " - "AggregationResult container (on a bootstrapped fit, " - "re-fit with n_bootstrap=0 or use the deprecated " - "fit-time aggregate=; a result unpickled from a pre-3.9 " - "release carries no kit and must be re-fit)." + "Group effects not stored on this fit. Use " + "results.aggregate('group') - analytical fits recompute " + "from the retained kit; bootstrapped fits relay the " + "surface computed at fit. A result unpickled from a " + "pre-3.9 release carries no kit and must be re-fit." ) rows = [] for g, data in sorted(self.group_effects.items()): diff --git a/diff_diff/visualization/_event_study.py b/diff_diff/visualization/_event_study.py index d73a53a7..fc79a61d 100644 --- a/diff_diff/visualization/_event_study.py +++ b/diff_diff/visualization/_event_study.py @@ -1067,7 +1067,8 @@ def _extract_plot_data( "SunAbrahamResults, ImputationDiDResults, " "ChaisemartinDHaultfoeuilleResults, EventStudyResults, or DataFrame. " "For a result that stores no event-study surface (DMLDiD, " - "EfficientDiD, every CallawaySantAnna fit since 4.0), plot the " + "EfficientDiD, every CallawaySantAnna fit since 4.0, and ANALYTICAL " + "ImputationDiD / TwoStageDiD / ContinuousDiD fits), plot the " "post-fit container: plot_event_study(results.aggregate(" "'event_study'))." ) diff --git a/docs/api/continuous_did.rst b/docs/api/continuous_did.rst index 5b762132..9aed4fa2 100644 --- a/docs/api/continuous_did.rst +++ b/docs/api/continuous_did.rst @@ -131,10 +131,10 @@ Accessing dose-response curves (always computed by ``fit()``):: Event study aggregation (post-fit, row M-025):: - # Dynamic effects (binarized ATT by relative period). The event-study - # recompute needs an ANALYTICAL fit - on a bootstrapped fit - # aggregate('event_study') raises; the deprecated fit-time - # aggregate='eventstudy' remains available for that case until 4.0. + # Dynamic effects (binarized ATT by relative period). On an analytical + # fit aggregate('event_study') recomputes from the retained kit; on a + # bootstrapped fit (n_bootstrap > 0) the percentile event study is + # computed at fit and aggregate('event_study') relays it (4.0). est_es = ContinuousDiD(seed=42) results_es = est_es.fit(data, outcome='outcome', unit='unit', time='period', first_treat='first_treat', diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 94221019..57d7c430 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1283,7 +1283,7 @@ The multiplier bootstrap uses random weights w_i with E[w]=0 and Var(w)=1: - **Note:** Repeated cross-sections (`panel=False`, Phase 7b): supports surveys like BRFSS, ACS annual, and CPS monthly where units are not followed over time. Uses cross-sectional DRDID (Sant'Anna & Zhao 2020, Section 4): `reg` matches `DRDID::reg_did_rc` (Eq 2.2), `dr` matches `DRDID::drdid_rc` (locally efficient, Eq 3.3+3.4 with 4 OLS fits), `ipw` matches `DRDID::std_ipw_did_rc`. Per-observation influence functions instead of per-unit. All three estimation methods support covariates and survey weights. - **Note:** Panel and RCS influence functions use the library-wide `phi_i = psi_i / n` convention (SE = `sqrt(sum(phi^2))`, algebraically equivalent to R's `sd(psi)*sqrt(n-1)/n`). Leading IF terms are computed on psi scale and divided by n; PS nuisance corrections are computed on psi scale (`score @ solve(Hessian)`) with a single `/n` conversion to phi. - **Note:** Non-survey DR path also includes nuisance IF corrections (PS + OR), matching the survey path structure (Phase 7a). Previously used plug-in IF only. As of v3.7 the non-survey reg and ipw paths carry their corrections too (OR estimation-effect / PS score), so the nuisance-IF treatment is method-uniform. -- **Note (post-fit aggregate() - rows M-020/M-117):** `fit(aggregate=, balance_e=)` was deprecated in 3.9 and REMOVED at 4.0 (PR 3b of the cut: `fit()` computes no event-study/group surface, so the legacy `event_study_effects`/`group_effects` fields - and `cband_crit_value`/`event_study_vcov`/`event_study_vcov_index`/`event_study_df` - are None after EVERY fit, `to_dataframe(level='event_study'|'group')` raises on fresh fits until the M-093 schema rewire, and `survey_design` is positional slot 7) in favor of post-fit `results.aggregate(type, balance_e=)` on the fit-retained aggregation kit ('simple' relays the stored overall inference; 'event_study'/'group' recompute from the kit — on BOOTSTRAPPED fits they REPLAY the fit-time multiplier bootstrap from the kit's `BootstrapReplaySpec` RNG-state snapshot: percentile se/CI/cband matching a fit-time aggregation to BLAS reassociation (~1 ULP, `assert_allclose`, never bit-identity; the discrete percentile p-value is a count statistic outside that claim), df cleared (no analytical provenance beside percentile inference), vcov/vcov_index from the same bootstrap draws (see the bootstrap event-study covariance Note above). **Note (df_survey two-channel contract):** the container-level `df_survey` SCALAR - the fit's resolved scalar inference df (`EventStudyResults.df_survey`) - DELIBERATELY persists on bootstrapped fit-time and replayed surfaces (CS, DMLDiD, EfficientDiD publish it identically): it is a live consumer channel (HonestDiD's container branches consume it as the `df=` for their critical values on the containers they admit), and only the PER-ROW `df` column is the per-row inference-provenance channel that percentile bootstrap clears. A bootstrapped ImputationDiD survey fit with `pretrends=True` clears the per-row channel even though the lead rows retained analytical survey-df inference (the bootstrap loop skips pre-period horizons) - the shipped producer convention, backend-stamped and failing closed on a Rust-vs-NumPy weight-backend mismatch or a pre-replay legacy pickle — the M-027 per-level policy, with the former blanket fail-closed retired). `DiagnosticReport` now derives the event-study container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends`, `pretrends_power`, and `sensitivity` checks run on every CS fit — CS containers are M-093-admitted into `compute_pretrends_power` / `HonestDiD.sensitivity_analysis` with pinned raw-route parity, and the raw field, when present (the requested-but-empty `{}` included), always takes precedence. `heterogeneity` is unaffected (it reads `group_time_effects` on plain fits). Bootstrapped fits now DERIVE successfully via the percentile-bootstrap replay: the derived container carries the covariance of the bootstrap draws as `vcov`, so `parallel_trends` runs the joint Wald test and `pretrends_power`/`sensitivity` use the full covariance (the Bonferroni and diagonal-covariance fallbacks remain where no bootstrap covariance is stored); the replay's re-emitted fit-time warnings are recorded and republished per section. The fail-closed skip remains for kit-less/legacy pickles, backend-mismatched replays, and the sibling estimators' bootstrap gates. The recompute levels' replay RE-EMITS the fit-time bootstrap warnings by design (the relay levels' no-re-warn convention is scoped to 'simple'/'total'). +- **Note (post-fit aggregate() - rows M-020/M-117):** `fit(aggregate=, balance_e=)` was deprecated in 3.9 and REMOVED at 4.0 (PR 3b of the cut: `fit()` computes no event-study/group surface, so the legacy `event_study_effects`/`group_effects` fields - and `cband_crit_value`/`event_study_vcov`/`event_study_vcov_index`/`event_study_df` - are None after EVERY fit, `to_dataframe(level='event_study'|'group')` raises on fresh fits until the M-093 schema rewire, and `survey_design` is positional slot 7) in favor of post-fit `results.aggregate(type, balance_e=)` on the fit-retained aggregation kit ('simple' relays the stored overall inference; 'event_study'/'group' recompute from the kit — on BOOTSTRAPPED fits they REPLAY the fit-time multiplier bootstrap from the kit's `BootstrapReplaySpec` RNG-state snapshot: percentile se/CI/cband matching a fit-time aggregation to BLAS reassociation (~1 ULP, `assert_allclose`, never bit-identity; the discrete percentile p-value is a count statistic outside that claim), df cleared (no analytical provenance beside percentile inference), vcov/vcov_index from the same bootstrap draws (see the bootstrap event-study covariance Note above). **Note (df_survey two-channel contract):** the container-level `df_survey` SCALAR - the fit's resolved scalar inference df (`EventStudyResults.df_survey`) - DELIBERATELY persists on bootstrapped replayed (CS, DMLDiD, EfficientDiD publish it identically) and relayed (ImputationDiD/TwoStageDiD/ContinuousDiD, 4.0 PR 3c) surfaces: it is a live consumer channel (HonestDiD's container branches consume it as the `df=` for their critical values on the containers they admit), and only the PER-ROW `df` column is the per-row inference-provenance channel that percentile bootstrap clears. A bootstrapped ImputationDiD survey fit with `pretrends=True` clears the per-row channel even though the lead rows retained analytical survey-df inference (the bootstrap loop skips pre-period horizons) - the shipped producer convention, backend-stamped and failing closed on a Rust-vs-NumPy weight-backend mismatch or a pre-replay legacy pickle — the M-027 per-level policy, with the former blanket fail-closed retired). `DiagnosticReport` now derives the event-study container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends`, `pretrends_power`, and `sensitivity` checks run on every CS fit — CS containers are M-093-admitted into `compute_pretrends_power` / `HonestDiD.sensitivity_analysis` with pinned raw-route parity, and the raw field, when present (the requested-but-empty `{}` included), always takes precedence. `heterogeneity` is unaffected (it reads `group_time_effects` on plain fits). Bootstrapped fits now DERIVE successfully via the percentile-bootstrap replay: the derived container carries the covariance of the bootstrap draws as `vcov`, so `parallel_trends` runs the joint Wald test and `pretrends_power`/`sensitivity` use the full covariance (the Bonferroni and diagonal-covariance fallbacks remain where no bootstrap covariance is stored); the replay's re-emitted fit-time warnings are recorded and republished per section. The fail-closed skip remains for kit-less/legacy pickles, backend-mismatched replays, and bootstrapped ImputationDiD/TwoStageDiD/ContinuousDiD fits whose fit-materialized surface was fail-soft cleared (since 4.0 PR 3c those three RELAY the surface computed at fit - see their bootstrapped-fit relay Notes). The recompute levels' replay RE-EMITS the fit-time bootstrap warnings by design (the relay levels' no-re-warn convention is scoped to 'simple'/'total'). - **Note (post-fit `aggregate('total')` - the estimator-owned total incremental outcome, 3.10):** (a) **Estimand:** `total = C x overall_att` with `se = C x overall_se`, CI scaled by `C`, `t`/`p`/`df` inherited unchanged - an exact relay CONDITIONAL ON THE REALIZED AGGREGATION MASS (never call it unqualified "exact"): `C` is the design-fixed complete-case treated-observation count on every admitted routing, the inherited SE prices the overall's full inference (WIF/estimated-share terms included) but treats `C` as fixed at its realized value, and the random-mass `att*dC` variance term is deliberately out of scope (the DEFERRED survey/RC remainder). Every SE branch is homogeneous of degree 1 in the scaling, so the relay is exact under analytical AND percentile-bootstrap inference. (b) **Fail-closed routing gates**, in order: (1) repeated-cross-section-routed fits (`panel=False`, or genuinely-unbalanced `allow_unbalanced_panel=True` fits - a balanced panel with the flag stays panel-routed and admitted) raise `NotImplementedError` - the RC cohort mass counts UNITS and weights every kept post period by it, a ~T-fold overcount of treated observations on a T-wave RCS (execution-verified); (2) fits declaring a `survey_design=` raise - the realized-mass relay omits the survey mass-uncertainty term and design-aware population-scale totals are not implemented (retained weight scale differs by design family: analytic pweight resolves normalized to sum = n and CS retains no PER-OBSERVATION raw-weight record - the `survey_metadata` scalars report raw-scale `sum_weights`/`weight_range` provenance, but not the per-obs raw vector a population-scale total would need; CS accepts pweight only - while replicate designs retain raw scale); the gate reads the fit-time `is_survey_fit` KIT snapshot, so bare-`cluster=` fits are ADMITTED (their synthesized all-ones internal design is not a survey) and a post-fit edit of the public `survey_metadata` cannot bypass it; the DECLARATION BOUNDARY is deliberate - an explicit unweighted `SurveyDesign(psu=...)` fit is numerically identical to bare `cluster=` yet fails closed by declaration (declaring survey_design opts into the survey contract; conservative, never wrong numbers; the message points at the `cluster=` route); a combined survey+RC fit deterministically gets the RC message; (3) the coincidence check: on bare-`cluster=` fits `_aggregate_simple` weights kept cells by the synthesized cohort masses (full cohort size per kept cell), and when kept cells have INCOMPLETE treated support that realized mass diverges from the complete-case count - the same overcount class as (1) - so the fit raises rather than publishing an ambiguous mass; clean bare-`cluster=` fits (the normal case) are admitted with `C` equal to both counts. Panel-routed pre-upgrade (legacy) kits missing the fit-time snapshots raise the refit message - never a mutable-field or public-replay fallback. (c) **Filter-replay contract:** `C` replays `_aggregate_simple`'s cell selection verbatim over the immutable `agg_gt_cells` fit-time kit snapshot (EDiD snapshot discipline; the kit builder stashes `(g, t, effect, n_treated)` 4-tuples in dict order) - anticipation filter FIRST (cells with `g - anticipation <= t < g` COUNT as post, the business-facing subtlety; universal-base reference cells are excluded by this filter alone, their base period being `< g - anticipation` by construction), finite-effect mask SECOND, weight = cohort mass where `fixed_cohort_agg_weights` provides one else per-cell complete-case `n_treated` (the source's `agg_weight` fallback is deliberately absent - RC-only, gated out), pairwise `np.sum`; the overall scalars come from the same stored fields the `'simple'` relay reads, so a post-fit mutation of `group_time_effects` moves NEITHER factor and `total == C x aggregate('simple').att` cannot desynchronize (pinned by a mutation test). NEVER use `agg_total_weight` (the RC all-units WIF mass). (d) **NaN rules:** empty post set / all-NaN effects give `C = NaN` and an all-NaN row WITHOUT re-emitting the fit-time UserWarnings (the RELAY-level no-re-warn convention — the NaN row is the signal; the recompute levels' bootstrap replay re-emits by design); with finite `C` the relay is VERBATIM - inherited NaNs (a degenerate fit's `(nan, nan)` CI beside finite att/se = 0) pass through, mirroring `aggregate('simple')` per the repo's non-estimable-row convention (att/n are never blanked by NaN inference fields); the ONLY additional blanking is true float overflow (a finite overall value made non-finite BY the `x C` scaling - unreachable in practice, guarded per this code family's finiteness convention). (e) **Bootstrap:** 'total' is a RELAY level - available on bootstrapped fits with the df column NaN (the M-027 per-level policy). (f) **Container conventions:** single row, `level="total"`, `label=["total"]`, `target=["total"]` (a total incremental outcome, NOT an ATT - the single-non-att neutral "estimate" rendering applies), `n = [C]` with `n_kind="obs"`, `weight=[1.0]`, per-class df carrier (CS: `resolve_inference_df` - finite on admitted bare-`cluster=` fits, NaN on plain analytical panel fits). Per-level `n` semantics are deliberately DIFFERENT: 'total''s `n` is the treated observations entering the support, while 'simple''s `n` stays treated+control units by contract - `n_kind` is per-container and each is documented, satisfying the never-conflate rule. (g) **MMM admission:** both `diff_diff.mmm` exporters accept the total container ALONE - `scale` (numeric or `"auto"`) is rejected as double-counting; for overall-total exports this route supersedes `scale="auto"` (see the MMM section). The total remains meaningful only for outcomes additive in levels - the same unverifiable caveat every scale route carries. @@ -1407,7 +1407,7 @@ Dynamic placebos `DID^{pl}_l` look backward from each group's reference period, - **Note (Phase 2 cost-benefit delta SE):** When `L_max >= 2`, `overall_att` holds the cost-benefit `delta`. Its SE is computed via the delta method from per-horizon SEs: `SE(delta) = sqrt(sum w_l^2 * SE(DID_l)^2)`, treating horizons as independent (conservative under Assumption 8). When bootstrap is enabled, per-horizon bootstrap SEs flow through the delta-method formula, so `overall_se` reflects bootstrap-derived per-horizon uncertainty but the delta aggregation itself uses normal-theory (not bootstrap percentile). This is an intentional exception to the general bootstrap-inference-surface contract: `overall_p_value` and `overall_conf_int` for `delta` use `safe_inference(delta, delta_se)`, not percentile bootstrap, because the delta is a derived aggregate rather than a directly bootstrapped estimand. -- **Note (post-fit `aggregate()` is a view - row M-026):** `results.aggregate('event_study')` and `results.aggregate('simple')` are pure VIEWS over stored fields - nothing is recomputed. `'event_study'` returns the unified `EventStudyResults` container (Phase-1 `L_max=None` fits: the 2-row l=1 view; `L_max >= 1`: the multi-horizon `l1_first_switch` surface); `'simple'` a one-row `AggregationResult` relaying `overall_att/se/t/p/CI` bit-exactly with the estimand-aware `target` label (DID_M / DID_1 / delta, or the trends-linear first-difference label whose overall row is all-NaN by design). Because it is a view, BOOTSTRAP FITS ARE PERMITTED - the library-wide per-level relay rule (since M-027 the recompute adopters' 'simple' relays are permitted on bootstrap fits too; their kit-based recompute levels REPLAY the fit-time multiplier bootstrap on CallawaySantAnna, DMLDiD (via the inherited CS kit channel), and EfficientDiD and stay closed on the remaining adopters - a recompute must never silently substitute analytical inference); here each row relays exactly the inference the fit stored, including the delta's analytical-with-df numbers under bootstrap per the `Note (Phase 2 cost-benefit delta SE)` above (the view's `df` column resolves from the actual inference path, not the bootstrap-cleared `event_study_df` channel). The dCDH event-study CONTAINER is deliberately rejected by `compute_honest_did`/`compute_pretrends_power` - its l1 placebo semantics need HonestDiD's native dCDH branch (mandatory reinterpretation warning + horizon trimming); pass the results object itself. +- **Note (post-fit `aggregate()` is a view - row M-026):** `results.aggregate('event_study')` and `results.aggregate('simple')` are pure VIEWS over stored fields - nothing is recomputed. `'event_study'` returns the unified `EventStudyResults` container (Phase-1 `L_max=None` fits: the 2-row l=1 view; `L_max >= 1`: the multi-horizon `l1_first_switch` surface); `'simple'` a one-row `AggregationResult` relaying `overall_att/se/t/p/CI` bit-exactly with the estimand-aware `target` label (DID_M / DID_1 / delta, or the trends-linear first-difference label whose overall row is all-NaN by design). Because it is a view, BOOTSTRAP FITS ARE PERMITTED - the library-wide per-level relay rule (since M-027 the recompute adopters' 'simple' relays are permitted on bootstrap fits too; their kit-based recompute levels REPLAY the fit-time multiplier bootstrap on CallawaySantAnna, DMLDiD (via the inherited CS kit channel), and EfficientDiD, while the three hybrids ImputationDiD/TwoStageDiD/ContinuousDiD RELAY the percentile surface materialized at fit since the fit-time keyword was removed at 4.0 (PR 3c) - a recompute must never silently substitute analytical inference); here each row relays exactly the inference the fit stored, including the delta's analytical-with-df numbers under bootstrap per the `Note (Phase 2 cost-benefit delta SE)` above (the view's `df` column resolves from the actual inference path, not the bootstrap-cleared `event_study_df` channel). The dCDH event-study CONTAINER is deliberately rejected by `compute_honest_did`/`compute_pretrends_power` - its l1 placebo semantics need HonestDiD's native dCDH branch (mandatory reinterpretation warning + horizon trimming); pass the results object itself. - **Note (dynamic placebo SE - library extension):** Dynamic placebos `DID^{pl}_l` (negative horizons in `placebo_event_study`) now have analytical SE and bootstrap SE when `L_max >= 1`. The placebo IF uses the same cohort-recentered structure as positive horizons, applied to backward outcome differences `Y_{g, F_g-1-l} - Y_{g, F_g-1}` with the dual-eligibility control pool (forward + backward observation required). The paper's Theorem 1 variance result is stated for `DID_l`, not `DID^{pl}_l` - this extension applies the same IF/variance structure to the placebo estimand as a library enhancement. The single-period placebo `DID_M^pl` (`L_max=None`) retains NaN SE because the per-period aggregation path has no IF derivation. @@ -1661,7 +1661,8 @@ labels.* 6. **Note (discrete-treatment saturated regression — library extension beyond `contdid` v0.1.0):** `treatment_type="discrete"` estimates the dose-response by a **saturated regression** (CGBS 2024 Eq. 4.1) — one indicator per distinct dose level, so `beta_j = mean_{D=d_j}(ΔY − control) = ATT(d_j)` (a per-level 2×2 DiD) — instead of the B-spline sieve. `ACRT(d_j)` is the paper's **backward difference** on the grid `{d_0 = 0, d_1, …, d_J}` (Eq. 4.1 makes `d_0 = 0` the omitted category with `ATT(0) = 0`): `ACRT(d_j) = [ATT(d_j) − ATT(d_{j-1})]/(d_j − d_{j-1})` for `j ≥ 2`, and at the lowest positive level it references the zero-dose baseline, `ACRT(d_1) = [ATT(d_1) − 0]/(d_1 − 0) = ATT(d_1)/d_1`. So a single positive dose (`J = 1`, e.g. binary `D ∈ {0,1}`) yields `ACRT(d_1) = ATT(d_1)/d_1`, and for `d_1 = 1` the documented binary identity `ACRT = ATT` holds exactly. This is a **library extension**: `contdid` v0.1.0 accepts `treatment_type` in its signature but **does not implement the discrete path** (documented "Discrete treatment not yet implemented"), so there is **no external R anchor**. It is instead an *exact* basis swap of the B-spline design/evaluation/derivative trio for an indicator/identity/finite-difference trio; every downstream quantity is linear in `beta`, so the analytical-SE / multiplier-bootstrap / covariate (reg,dr) / survey machinery is reused unchanged and reduces *analytically* to the per-level 2×2 DiD (`bread @ psi_bar = ones(J)`; the common control mean cancels in the `j ≥ 2` adjacent differences whose `L`-rows sum to 0). **reg vs dr:** the constant DR augmentation `η̄_cont` cancels in the `j ≥ 2` differences, so `ACRT(d_j)` point AND SE are identical for `reg`/`dr` there; but `ACRT(d_1) = ATT(d_1)/d_1` references the fixed baseline `ATT(0) = 0` (not shifted by `η̄_cont`), so `reg` and `dr` genuinely **differ at `ACRT(d_1)` by `η̄_cont/d_1`** (and correspondingly in `ACRT^glob` via the `d_1` mass) — the dr influence function carries the augmentation variance at `d_1` (validated: analytical `ACRT(d_1)` SE matches the multiplier bootstrap). Validation (R-free, in CI): exact hand-calc of `ATT(d_j)`/`ACRT`/`overall_att` and the analytical SE against a direct per-level 2×2 reconstruction (`~1e-12`/`~1e-10`), DGP recovery, and MC coverage for analytical + bootstrap (`tests/test_methodology_continuous_did.py::TestDiscreteSaturated`, `tests/test_continuous_did.py::TestDiscreteSaturatedAPI`). **Fail-closed policies (no-silent-failures):** (i) multi-cohort fits with **heterogeneous dose support** across cohorts raise `NotImplementedError` — an absent global level yields a dropped zero column (`att_d[level]=0`) that the plain-sum dose aggregation would bias toward zero (support-aware aggregation is deferred; single-cohort, 2-period, and shared-support multi-cohort are supported); (ii) a requested `dvals` value that is not an observed dose level raises `ValueError` (a saturated model cannot be evaluated off-support); (iii) an over-parameterized fit (`< 2` treated units per level, or `J > n_treated/2`) warns (degenerate per-level SE); (iv) with `survey_design=`, any dose level with **zero effective treated mass in a `(g,t)` cell** raises `ValueError` — a per-cell check (not just the global positive-weight check), so a level that survey/subpopulation weights zero out for one cohort while another cohort keeps it cannot silently drop to a zero-coefficient saturated column. Cross-references `docs/methodology/continuous-did.md` § 5.1. 7. **Note (lowest-dose-as-control, Remark 3.1 — library extension beyond `contdid` v0.1.0):** `control_group="lowest_dose"` implements CGBS 2024 Remark 3.1 for settings with no untreated group (`P(D=0) = 0`): the lowest-dose group `d_L` becomes the comparison and the estimand is `ATT(d) − ATT(d_L)` (SPT), with `ATT(d_L) = 0` the omitted reference. Mechanically it is a **control-group swap** — the D=0 control pool is replaced by the `d_L` group; the entire linear influence-function / bootstrap / event-study / survey machinery is control-group-generic and reused unchanged (`ee_control` already carries the reference-group variance, so **no new SE plumbing**). On the discrete saturated basis the backward-difference operator's reference shifts from `0` to `d_L` (`ACRT(d_1) = ATT(d_1)/(d_1 − d_L)`); on the continuous B-spline path the reference shifts only `μ_0` (the level), leaving `ACRT = spline'` unchanged. `contdid` v0.1.0 does **not** implement Remark 3.1, so there is **no external R anchor**; validation (R-free, in CI): an **exact `d_L → 0` equivalence** anchor (relabelling a `never_treated` panel's D=0 group as a tiny common dose `d_L = ε` reproduces the `never_treated` ATT and SE exactly, for any ε), a discrete hand-calc of `ATT(d)−ATT(d_L)`/`ACRT`/`overall_att`/`overall_acrt` and the per-level 2×2 SE (`~1e-10`), continuous mass-point DGP recovery, analytical-vs-bootstrap SE agreement, a pre-period placebo, and MC coverage (`tests/test_methodology_continuous_did.py::TestLowestDose`, `tests/test_continuous_did.py::TestLowestDoseAPI`). The continuous path requires a genuine **mass point** at the minimum dose (`>= 2` units at `d_L`, i.e. `P(D=d_L) > 0`) — the Remark 3.1 identification condition; a singleton minimum fails closed. **Fail-closed policies (no-silent-failures):** (i) never-treated units present with `lowest_dose` → `ValueError` (they would be silently dropped); (ii) singleton `d_L` (no mass point) → `ValueError`; (iii) no treated dose above `d_L` → `ValueError`; (iv) user `dvals ≤ d_L` → `ValueError` (`d_L` is the omitted reference); (v) survey/subpopulation weighting that leaves the `d_L` group with `< 2` positive-weight units → `ValueError` (a single positive-weight reference unit gives `ee_control = 0`, i.e. zero control-side variance — the effective-`>= 2` analogue of the raw mass-point guard, applied after weighting); (vi) a boundary gap `d_1 − d_L` that is a tiny fraction of the dose range warns (huge boundary ACRT/SE). **Deferred (fail-closed `NotImplementedError` + TODO):** multi-cohort `lowest_dose` (needs a within-cohort reference + support-aware cross-cohort aggregation) and `covariates=` × `lowest_dose` (conditional-PT-relative-to-`d_L` estimand). Cross-references `docs/methodology/continuous-did.md` § 5.6. -8. **Note (post-fit aggregate() - rows M-025/M-122):** `fit(aggregate=)` is deprecated in 3.9 (removed in 4.0; the no-underscore `"eventstudy"` spelling dies with it) in favor of post-fit `results.aggregate(type)` on the unified vocabulary + `'dose'` as this estimator's documented extra level; the PRE-EXISTING fit-time value validation is retained (unknown strings still raise `ValueError` after the deprecation warning - unlike the EfficientDiD/Imputation shims, which never validated). (a) **MIXED view/recompute architecture** (unique among the aggregate-postfit adopters): the dose-response curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by `fit()` (`aggregate="dose"` was a fit-time no-op), so `aggregate('simple')` (2 rows, targets att/acrt - the dual-estimand case the `AggregationResult.target` column exists for; `n` = the DISJOINT treated+control unit total, `n_kind='units'`) and `aggregate('dose')` (2N target-discriminated rows; labels = the dose grid twice; no count/mass per row) are pure VIEWS over stored public fields, PERMITTED on bootstrap fits (the library-wide per-level relay rule, since M-027 converged CS/EDiD/Imputation/TwoStage onto it) - they relay stored inference verbatim, including the FINITE `safe_inference` t-stat fit stores beside the percentile p/CI on bootstrapped overall rows and the `DoseResponseCurve.to_dataframe`-exact NaN-t derivation on dose rows; only the df column is uniformly NaN under bootstrap. (b) **`aggregate('event_study')` recomputes** the binarized event study from a pruned per-cell IF payload retained on the fit-built kit: per-(g,t) treated/control positional indices, `delta_y_treated`, `ee_control`, masses and the covariate-path `if_att_glob` (O(n_treated+n_control) per cell), unit-level arrays, and - on survey fits - the PANEL-LEVEL `ResolvedSurveyDesign` ref (the recompute performs the unit collapse itself, keeping the moved body verbatim; on replicate designs the (n_obs x R) replicate matrix rides along - the documented memory cost). The K-dimensional spline machinery (bread, `ee_treated`, `Psi_eval`, `dPsi_*`) is NOT retained; no panel data columns and no raw unit identifiers are retained. Replicate-weight designs ARE supported post-fit (IF-based `compute_replicate_if_variance` - no refit replay). Bootstrap fits carry a SCALARS-ONLY kit and the event-study route fails closed (`NotImplementedError`; the deprecated fit-time `aggregate='eventstudy'` computes the bootstrap surface until 4.0, or re-fit with `n_bootstrap=0`) - a seeded post-fit bootstrap replay is the TODO.md row. (c) **Fit-faithful quirk:** when no post-treatment (g,t) cells exist, event-study rows keep NaN inference on BOTH routes (the fit-time surface never fills them). (d) **df provenance:** the stored `dose_response_att.df_survey` channel (the value every fit-time `safe_inference` received) drives the views' df column (finite-and-positive else NaN - the replicate-undefined 0 sentinel reports NaN in the column but feeds the t/p derivation raw); the event-study container's PER-ROW df column now threads the same survey df via the results-level `event_study_df` channel (M-092 completion; finite on analytical survey fits, None on non-survey/bootstrapped/no-surface fits and for the 0 sentinel), beside the scalar `df_survey` channel (the two-channel contract - see the CallawaySantAnna post-fit Note). (e) **Rendering:** this is the FIRST heterogeneous-`target` `AggregationResult`; `summary()`/`to_dataframe()` gained the target column / first-appearance target-block ordering amendment (normative rule in `docs/v4-design.md` section 6; uniform-target producers byte-stable). (f) **Consumer admission:** `compute_honest_did`/`compute_pretrends_power` reject ContinuousDiD containers BY DESIGN - no joint event-study covariance exists (per-bin IF SEs only) and the binarized bins carry no reference-period normalization at all (see M-093). Warning stacklevels in the moved bodies remain tuned for the fit-time frame depth, so post-fit-route warnings attribute to a library frame (the shipped EfficientDiD convention). (g) **Report-layer consumption:** `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent) for its `heterogeneity` check — the only ES-gated check in ContinuousDiD's applicability; captured recompute warnings are re-published on the section and derivation failures (bootstrap fits) surface as explicit skip context. +8. **Note (post-fit aggregate() - rows M-025/M-122):** `fit(aggregate=)` was deprecated in 3.9 and REMOVED at 4.0 (PR 3c of the cut; the no-underscore `"eventstudy"` spelling and the fit-time value validation died with it; `survey_design`/`covariates` are positional slots 7/8) in favor of post-fit `results.aggregate(type)` on the unified vocabulary + `'dose'` as this estimator's documented extra level. (a) **MIXED view/recompute architecture** (unique among the aggregate-postfit adopters): the dose-response curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by `fit()` (`aggregate="dose"` was a fit-time no-op), so `aggregate('simple')` (2 rows, targets att/acrt - the dual-estimand case the `AggregationResult.target` column exists for; `n` = the DISJOINT treated+control unit total, `n_kind='units'`) and `aggregate('dose')` (2N target-discriminated rows; labels = the dose grid twice; no count/mass per row) are pure VIEWS over stored public fields, PERMITTED on bootstrap fits (the library-wide per-level relay rule, since M-027 converged CS/EDiD/Imputation/TwoStage onto it) - they relay stored inference verbatim, including the FINITE `safe_inference` t-stat fit stores beside the percentile p/CI on bootstrapped overall rows and the `DoseResponseCurve.to_dataframe`-exact NaN-t derivation on dose rows; only the df column is uniformly NaN under bootstrap. (b) **`aggregate('event_study')` recomputes** the binarized event study from a pruned per-cell IF payload retained on the fit-built kit: per-(g,t) treated/control positional indices, `delta_y_treated`, `ee_control`, masses and the covariate-path `if_att_glob` (O(n_treated+n_control) per cell), unit-level arrays, and - on survey fits - the PANEL-LEVEL `ResolvedSurveyDesign` ref (the recompute performs the unit collapse itself, keeping the moved body verbatim; on replicate designs the (n_obs x R) replicate matrix rides along - the documented memory cost). The K-dimensional spline machinery (bread, `ee_treated`, `Psi_eval`, `dPsi_*`) is NOT retained; no panel data columns and no raw unit identifiers are retained. Replicate-weight designs ARE supported post-fit (IF-based `compute_replicate_if_variance` - no refit replay). Bootstrap fits carry a SCALARS-ONLY kit; since 4.0 (PR 3c) they ALWAYS materialize the binarized event study at fit and `aggregate('event_study')` RELAYS it (see the bootstrapped-fit relay Note below) - a genuine seeded post-fit bootstrap replay is the TODO.md row. (c) **Fit-faithful quirk:** when no post-treatment (g,t) cells exist, event-study rows keep NaN inference on BOTH routes (the fit-time surface never fills them). (d) **df provenance:** the stored `dose_response_att.df_survey` channel (the value every fit-time `safe_inference` received) drives the views' df column (finite-and-positive else NaN - the replicate-undefined 0 sentinel reports NaN in the column but feeds the t/p derivation raw); the event-study container's PER-ROW df column now threads the same survey df via the results-level `event_study_df` channel (M-092 completion; finite on analytical survey fits, None on non-survey/bootstrapped/no-surface fits and for the 0 sentinel), beside the scalar `df_survey` channel (the two-channel contract - see the CallawaySantAnna post-fit Note). (e) **Rendering:** this is the FIRST heterogeneous-`target` `AggregationResult`; `summary()`/`to_dataframe()` gained the target column / first-appearance target-block ordering amendment (normative rule in `docs/v4-design.md` section 6; uniform-target producers byte-stable). (f) **Consumer admission:** `compute_honest_did`/`compute_pretrends_power` reject ContinuousDiD containers BY DESIGN - no joint event-study covariance exists (per-bin IF SEs only) and the binarized bins carry no reference-period normalization at all (see M-093). Warning stacklevels in the moved bodies remain tuned for the fit-time frame depth, so post-fit-route warnings attribute to a library frame (the shipped EfficientDiD convention). (g) **Report-layer consumption:** `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent) for its `heterogeneity` check — the only ES-gated check in ContinuousDiD's applicability; captured recompute warnings are re-published on the section; bootstrapped fits are read from the fit-materialized raw field (raw-field precedence), and derivation failures (kit-less pickles, a fail-soft-cleared bootstrapped surface) surface as explicit skip context. +- **Note (bootstrapped-fit relay, 4.0):** (i) RELAY: a fit with `n_bootstrap > 0` ALWAYS computes the binarized event study at fit (the only route - the kit stays SCALARS-ONLY there, so no recompute is ever possible on a bootstrapped fit and no new kit key is added: a 3.9-3.13 pickle must not `KeyError`), stores it in `event_study_effects` (percentile se/p/CI, `event_study_df=None`), and `aggregate('event_study')` RELAYS it (alpha/anticipation/`base_period`/survey metadata from the kit snapshot; per-row `df` column NaN; `df_survey` scalar from the carrier's survey metadata; `vcov`/`vcov_index` None; no reference row). The gate stays the kit scalar `n_bootstrap > 0` (not the mutable public field); `balance_e` applies to no level here (the mixin rejects it). Analytic fits are unchanged (kit recompute; legacy field None) - the dead analytic fit-time inference arm was deleted. (ii) FAIL-SOFT materialization + storage rule: the surface is built inside one guard - any exception leaves it None and `_run_bootstrap` runs for the overall/dose targets only (the pre-4.0 plain-fit path); a `_run_bootstrap` attempt that raises (anything but the replicate-design `NotImplementedError`/`ValueError`, which are re-raised - the check fires inside `_run_bootstrap` after the ES is computed) WITH the surface present clears it and RETRIES ONCE overall-only (`np.random.default_rng(self.seed)` is rebuilt on entry, so a seeded retry reproduces attempt 1's weight draws and the overall/dose quintets equal a plain fit's); a retry failure propagates out of `fit()` exactly as before 4.0, so a RETURNED bootstrapped fit always carries bootstrap inference. STORAGE RULE: the surface is stored IFF `_run_bootstrap` ran (post-treatment cells exist); a bootstrapped fit with no post cells clears it and `aggregate('event_study')` raises the shared absent-surface `ValueError` (before 4.0 that fit raised `NotImplementedError` from the bootstrap gate; its overall ATT is already NaN with a `UserWarning`). The fail-soft `UserWarning` fires only when the fit carries bootstrap inference and the surface was dropped: 'ContinuousDiD: the event-study/group surfaces could not be materialized on this bootstrapped fit ({e}); the bootstrap covered the overall ATT only, so aggregate('event_study') / aggregate('group') are unavailable here - re-fit with n_bootstrap=0 for the analytical post-fit recompute.'; the relay then raises 'This bootstrapped ContinuousDiDResults carries no stored event_study surface: either it was fitted before 4.0 (a bootstrapped plain fit built none) or the surface could not be materialized at fit (see the fit-time warning). Re-fit with diff-diff >= 4.0 for the relayed surface, or with n_bootstrap=0 for the analytical post-fit recompute.' Net invariant: a bootstrapped plain fit's overall/dose quintets and success/abort outcome are unchanged by the 4.0 removal. (iii) The ES aggregator's `UserWarning`s and the per-target bootstrap-stats `RuntimeWarning`s ('Dropping n/N non-finite bootstrap samples in ... event study (h=...)', 'Too few valid bootstrap samples', 'Bootstrap SE is non-finite or zero') now fire on bootstrapped plain fits that never emitted them. (iv) Pre-4.0 pickles: a 3.9-3.13 bootstrapped pickle fitted with `fit(aggregate='eventstudy')` relays its stored surface; one fitted plain stores none and the relay raises the refit `ValueError`. (v) The relay reads the MUTABLE public `event_study_effects` field (a view by definition - the StackedDiD precedent); only alpha/anticipation/base_period/survey metadata come from kit snapshots, so the recompute route's 'kit snapshots only' invariant does not hold for the relayed dict. (vi) Downstream: `summary()` prints the event-study table and `to_dataframe(level='event_study')` returns a frame on bootstrapped plain fits; `DiagnosticReport`/`BusinessReport` run `heterogeneity` (the only ES-gated check here; no sensitivity) on the relayed surface instead of skipping with `NotImplementedError`; `plot_event_study(results)` plots it natively. 9. **Note:** `survey_metadata` is recomputed from the UNIT-level resolved design on EVERY inference arm - analytic, bootstrap, and the degenerate no-post-cells arm - matching the CallawaySantAnna/EfficientDiD convention (3.12; previously the bootstrap arm published the obs-level metadata from `_resolve_survey_for_fit`, so `sum_weights`/`effective_n`/`n_psu` - and, on implicit-PSU designs, `df_survey` - diverged from the analytic arm by panel length). Provenance-only: estimates and inference are unaffected (the bootstrap arm's inference remains percentile with no df). ### Implementation Checklist @@ -2173,14 +2174,14 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. inert-config warning predicate is REACHABILITY-based, no longer aggregate-keyed: an explicitly non-default value warns iff the per-lead inference is unreachable for the fit config — reachable iff - `pretrends=True` AND not a replicate-weight design AND (the deprecated - fit-time `aggregate='event_study'/'all'` was supplied OR - `n_bootstrap <= 0`, since post-fit `results.aggregate('event_study')` - reaches the leads on any analytical pretrends fit but fails closed on - bootstrapped ones). Reachability-BASED, not exact: a fit whose bootstrap - later FAILS (`bootstrap_results=None`) can still aggregate post-fit, so - that corner warns spuriously — the warning fires before the bootstrap - runs and cannot know. + `pretrends=True` AND not a replicate-weight design. Since 4.0 (PR 3c, + the fit-time `aggregate=` removal) the predicate is EXACT: analytical + pretrends fits reach the leads through post-fit + `results.aggregate('event_study')`, and bootstrapped pretrends fits + materialize the lead inference at fit (relayed by `aggregate()`), so + the former bootstrap spurious-warning corner no longer exists; the + replicate-weight exclusion stands because the per-replicate lead refits + are unimplemented (`aggregate('event_study')` raises there). *Edge cases:* - **Unbalanced panels:** FE estimated via iterative alternating projection (Gauss-Seidel), equivalent to OLS with unit+time dummies. Converges in O(max_iter) passes; typically 5-20 iterations for unbalanced panels, 1-2 for balanced. One-pass demeaning is only exact for balanced panels. @@ -2225,7 +2226,8 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. - [x] Supports balanced and unbalanced panels (iterative Gauss-Seidel demeaning for exact FE) - [x] Event study and group aggregation -- **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `imputation_did()` wrapper that used to forward the shared sentinel was removed at 4.0 per M-070 - PR 2 of the cut) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (the per-target psi machinery makes exact replay tractable - a TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the PER-ROW df column now threads the FINAL survey df via the results-level `event_study_df` channel - M-092 completion; `survey_df_final` at fit, the level-matched replay value on replicate replays; leads included on survey fits; None on non-survey/bootstrapped/no-surface fits and for the 0 sentinel - beside the scalar `df_survey` channel; see the CS post-fit Note's two-channel contract). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over, of which only finite-tau-hat observations enter the average - `n` reports the raw count, so on partially unidentified fits `n` exceeds the averaged support) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits) and `heterogeneity` checks run without the deprecated fit-time kwarg; the re-emitted recompute warnings are captured and re-published on the consuming report section (record-and-republish), and derivation failures (bootstrap, missing kit, the replicate gate above) surface as explicit per-check skip reasons. (f) `aggregate('total')` (3.10): the estimator-owned total incremental outcome - an exact relay `C x overall` CONDITIONAL on the realized aggregation mass, with `C` the FINITE-tau complete-case support (`total_support`, a float SNAPSHOT stashed at kit build precisely because the kit's `df` is a live `_fit_data` reference - a post-fit frame edit must not move the total; pre-upgrade kits missing the stash raise the refit message). Since the overall IS the finite-support mean, `C x overall = sum(tau)` exactly - this total uses the FINITE support and thereby fixes the documented raw-|Omega_1| overcount of the MMM `scale=\"auto\"` route for total exports ('simple''s `n` stays the raw count by contract; different levels, different documented n semantics). Zero support (the all-unidentified fit, overall already NaN) -> NaN mass -> all-NaN row (no re-warn); finite `C` relays verbatim - the constant-outcome degenerate fit keeps finite att/se = 0 beside its inherited `(nan, nan)` CI, mirroring 'simple'. Unweighted fits only: declared `survey_design=` fails closed via the kit's `survey_metadata` snapshot (mutation-proof). Bootstrap fits relay with a NaN df column (M-027); df carrier = `survey_df_final`; container: single `target='total'` row, `n_kind='obs'`. MMM: admitted with NO scale (see the MMM section). +- **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` was deprecated in 3.9 and REMOVED at 4.0 (PR 3c of the cut; `survey_design` is positional slot 7; the `imputation_did()` wrapper that used to forward the shared sentinel was removed per M-070 - PR 2) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column; since 4.0 (PR 3c) the ES/group levels RELAY the percentile surface materialized at fit (see the bootstrapped-fit relay Note below; `balance_e` fails closed there - the balanced-window replay via the retained psi machinery is the TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the PER-ROW df column now threads the FINAL survey df via the results-level `event_study_df` channel - M-092 completion; `survey_df_final` at fit, the level-matched replay value on replicate replays; leads included on survey fits; None on non-survey/bootstrapped/no-surface fits and for the 0 sentinel - beside the scalar `df_survey` channel; see the CS post-fit Note's two-channel contract). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over, of which only finite-tau-hat observations enter the average - `n` reports the raw count, so on partially unidentified fits `n` exceeds the averaged support) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises NotImplementedError (per-replicate lead refits unimplemented; the unreachable fit-time twin of this gate was deleted at 4.0); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits) and `heterogeneity` checks run on analytic plain fits (bootstrapped fits are read from the fit-materialized raw fields - raw-field precedence); the re-emitted recompute warnings are captured and re-published on the consuming report section (record-and-republish), and derivation failures (missing kit, the replicate gate above, a fail-soft-cleared bootstrapped surface) surface as explicit per-check skip reasons. (f) `aggregate('total')` (3.10): the estimator-owned total incremental outcome - an exact relay `C x overall` CONDITIONAL on the realized aggregation mass, with `C` the FINITE-tau complete-case support (`total_support`, a float SNAPSHOT stashed at kit build precisely because the kit's `df` is a live `_fit_data` reference - a post-fit frame edit must not move the total; pre-upgrade kits missing the stash raise the refit message). Since the overall IS the finite-support mean, `C x overall = sum(tau)` exactly - this total uses the FINITE support and thereby fixes the documented raw-|Omega_1| overcount of the MMM `scale=\"auto\"` route for total exports ('simple''s `n` stays the raw count by contract; different levels, different documented n semantics). Zero support (the all-unidentified fit, overall already NaN) -> NaN mass -> all-NaN row (no re-warn); finite `C` relays verbatim - the constant-outcome degenerate fit keeps finite att/se = 0 beside its inherited `(nan, nan)` CI, mirroring 'simple'. Unweighted fits only: declared `survey_design=` fails closed via the kit's `survey_metadata` snapshot (mutation-proof). Bootstrap fits relay with a NaN df column (M-027); df carrier = `survey_df_final`; container: single `target='total'` row, `n_kind='obs'`. MMM: admitted with NO scale (see the MMM section). +- **Note (bootstrapped-fit relay, 4.0):** (i) RELAY: a fit with `n_bootstrap > 0` ALWAYS computes the event-study AND group percentile surfaces at fit (`balance_e=None` - exactly what the 3.x `fit(aggregate='all')` computed; percentile se/p/CI, recomputed t, `df_used`/`event_study_df` cleared), stores them in `event_study_effects`/`group_effects`, and `aggregate('event_study')`/`aggregate('group')` RELAY them (alpha/anticipation/survey metadata from the kit snapshot; per-row `df` column NaN - the `pretrends=True` lead rows show NaN df too, identical to the 3.x bootstrapped surface, by design; `df_survey` scalar from the carrier's survey metadata; `vcov`/`vcov_index` None; reference rows via the zero-count sentinel). `aggregate('event_study', balance_e=k)` on a bootstrapped fit FAILS CLOSED (`NotImplementedError`: the stored surface is unrestricted and the per-target draws are not retained; re-fit with `n_bootstrap=0`). Analytic fits are unchanged (kit recompute; legacy fields None). The gate is `bootstrap_results is not None`: `None` (n_bootstrap == 0, OR a bootstrapped fit that ended analytical) means the analytical kit RECOMPUTE, the pre-4.0 behaviour. (ii) FAIL-SOFT materialization + storage rule: the surfaces are built inside one guard - any exception (e.g. the `pretrends=True` + `rank_deficient_action='error'` lead regression raising `ValueError` on a lead-collinear covariate) leaves them None and the bootstrap runs overall-only (the pre-4.0 plain-fit path, which SUCCEEDS where the 3.x `fit(aggregate='event_study')` raised); a bootstrap attempt that raises (anything but `NotImplementedError`, re-raised) WITH surfaces present clears them and RETRIES ONCE overall-only - `_precompute_bootstrap_psi` + `_run_bootstrap` re-run with `event_study_effects=None, group_effects=None`; `np.random.default_rng(self.seed)` is rebuilt on entry, so a seeded retry reproduces attempt 1's weight draws and the overall quintet equals a plain fit's (an unseeded retry draws a fresh, equally valid realization). The retry keeps the pre-4.0 failure semantics: a psi-precompute failure -> the 'Bootstrap pre-computation failed ... Skipping bootstrap inference.' warning + analytical overall; a `_run_bootstrap` failure -> RE-RAISED (the fit aborts exactly as before). STORAGE RULE: the legacy fields hold a surface IFF `bootstrap_results is not None`; when the fit ends analytical (`n_valid == 0` skipped the psi precompute, total failure) they are cleared and `aggregate()` RECOMPUTES analytically - an analytical surface must never sit on a bootstrapped fit where `summary()`/`to_dataframe()`/the DiagnosticReport raw-field precedence would read it as the fit's inference. The fail-soft `UserWarning` fires only when the fit DOES carry bootstrap inference and a surface was dropped: 'ImputationDiD: the event-study/group surfaces could not be materialized on this bootstrapped fit ({e}); the bootstrap covered the overall ATT only, so aggregate('event_study') / aggregate('group') are unavailable here - re-fit with n_bootstrap=0 for the analytical post-fit recompute.'; the relay then raises the shared absent-surface `ValueError` ('This bootstrapped ImputationDiDResults carries no stored event_study surface: either it was fitted before 4.0 (a bootstrapped plain fit built none) or the surface could not be materialized at fit (see the fit-time warning). Re-fit with diff-diff >= 4.0 for the relayed surface, or with n_bootstrap=0 for the analytical post-fit recompute.'). Net invariant: a bootstrapped plain fit's overall quintet and success/abort outcome are unchanged by the 4.0 removal. (iii) Newly reachable warnings on bootstrapped plain fits: the aggregators' own `UserWarning`s (Prop-5 unidentified horizons, empty horizon after filters, LOO single-unit auxiliary groups) and the per-target bootstrap-stats `RuntimeWarning`s ('Dropping n/N non-finite bootstrap samples in ... event study (h=...)', 'Too few valid bootstrap samples', 'Bootstrap SE is non-finite or zero'). (iv) Pre-4.0 pickle limitation: a 3.9-3.13 bootstrapped pickle fitted with `fit(aggregate=..., balance_e=k)` stores a balance_e-RESTRICTED surface and records no fit-time `balance_e`, so the relay returns that surface as stored - re-fit under 4.0 for the unrestricted one; one fitted plain stores none and the relay raises the refit `ValueError`. (v) The relay reads the MUTABLE public `event_study_effects`/`group_effects` fields (a view by definition - the StackedDiD precedent); only alpha/anticipation/survey metadata come from kit snapshots, so the recompute route's 'kit snapshots only' invariant does not hold for the relayed dict. (vi) Downstream: `summary()` prints the ES/group tables and `to_dataframe(level='event_study'|'group')` returns frames on bootstrapped plain fits; `DiagnosticReport`/`BusinessReport` run `heterogeneity` (on the relayed GROUP percentile dict - `_collect_effect_scalars` reads `group_effects` first) and `parallel_trends` (`pretrends=True`) on the relayed surface instead of skipping with `NotImplementedError` (no sensitivity check applies); `plot_event_study(results)` plots them natively. Replicate-weight designs never meet the always-on surfaces (replicate + `n_bootstrap > 0` is rejected before them), so the fit-time replicate stack is always `[overall]`-only - the degenerate-design overall-row migration delta in (e) stands. - **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- @@ -2319,7 +2321,8 @@ Our implementation uses multiplier bootstrap on the GMM influence function: clus - [x] Multiplier bootstrap on GMM influence function - [x] Event study and overall ATT aggregation -- **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `two_stage_did()` wrapper that used to forward the shared sentinel was removed at 4.0 per M-071 - PR 2 of the cut) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class; the pre-filter |Omega_1| count, while the ATT's Stage-2 support excludes rows whose `y_tilde` is non-finite - on such degenerate fits `n` exceeds the averaged support) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits; the recomputed joint vcov drives the joint-Wald path) and `heterogeneity` checks run without the deprecated fit-time kwarg; re-emitted recompute warnings are captured and re-published on the consuming report section, and derivation failures surface as explicit per-check skip reasons. (f) `aggregate('total')` (3.10): the estimator-owned total incremental outcome - an exact relay `C x overall` CONDITIONAL on the realized aggregation mass, with `C` the POST-FILTER treatment-indicator support the second stage actually regressed on (`omega_1_mask` minus non-finite `_y_tilde` rows, computed inline from the kit's private column-subset frame COPY - snapshot-grade, no fit-time stash needed, and the fit-time masker is never re-called so its warnings are not re-emitted; `score_pad_mask` is irrelevant here - it pads scores/variance only). This is the post-filter count 'simple''s raw pre-filter `n` documents as exceeding on degenerate fits. `D.sum() == 0` -> NaN mass -> all-NaN row (mirrors the fit's `(nan, nan)`); finite `C` relays verbatim. Unweighted fits only: declared `survey_design=` fails closed via the kit's `survey_metadata` snapshot (mutation-proof). Bootstrap fits relay with a NaN df column (M-027); df carrier = `survey_df_final`; container: single `target='total'` row, `n_kind='obs'`. MMM: admitted with NO scale (see the MMM section). +- **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` was deprecated in 3.9 and REMOVED at 4.0 (PR 3c of the cut; `survey_design` is positional slot 7; the `two_stage_did()` wrapper that used to forward the shared sentinel was removed per M-071 - PR 2) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while since 4.0 (PR 3c) the ES/group levels RELAY the percentile surface materialized at fit (see the bootstrapped-fit relay Note below; `balance_e` fails closed there - a balanced-window replay needs retained per-level GMM scores, which are function-locals: the TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally (analytical recompute). (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class; the pre-filter |Omega_1| count, while the ATT's Stage-2 support excludes rows whose `y_tilde` is non-finite - on such degenerate fits `n` exceeds the averaged support) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits; the recomputed joint vcov drives the joint-Wald path) and `heterogeneity` checks run on analytic plain fits (bootstrapped fits are read from the fit-materialized raw fields - raw-field precedence, no joint vcov there); re-emitted recompute warnings are captured and re-published on the consuming report section, and derivation failures (missing kit, a fail-soft-cleared bootstrapped surface) surface as explicit per-check skip reasons. (f) `aggregate('total')` (3.10): the estimator-owned total incremental outcome - an exact relay `C x overall` CONDITIONAL on the realized aggregation mass, with `C` the POST-FILTER treatment-indicator support the second stage actually regressed on (`omega_1_mask` minus non-finite `_y_tilde` rows, computed inline from the kit's private column-subset frame COPY - snapshot-grade, no fit-time stash needed, and the fit-time masker is never re-called so its warnings are not re-emitted; `score_pad_mask` is irrelevant here - it pads scores/variance only). This is the post-filter count 'simple''s raw pre-filter `n` documents as exceeding on degenerate fits. `D.sum() == 0` -> NaN mass -> all-NaN row (mirrors the fit's `(nan, nan)`); finite `C` relays verbatim. Unweighted fits only: declared `survey_design=` fails closed via the kit's `survey_metadata` snapshot (mutation-proof). Bootstrap fits relay with a NaN df column (M-027); df carrier = `survey_df_final`; container: single `target='total'` row, `n_kind='obs'`. MMM: admitted with NO scale (see the MMM section). +- **Note (bootstrapped-fit relay, 4.0):** (i) RELAY: a fit with `n_bootstrap > 0` ALWAYS computes the event-study (`_stage2_event_study`) AND group (`_stage2_group`) percentile surfaces at fit (`balance_e=None` - exactly what the 3.x `fit(aggregate='all')` computed; percentile se/p/CI; `event_study_vcov`/`event_study_vcov_index`/`event_study_df` None), stores them in `event_study_effects`/`group_effects`, and `aggregate('event_study')`/`aggregate('group')` RELAY them (alpha/anticipation/survey metadata from the kit snapshot; per-row `df` column NaN on both levels - the group relay passes `group_df=None` so no analytical df leaks beside percentile inference; `df_survey` scalar from the carrier's survey metadata; `vcov`/`vcov_index` None; reference rows via the zero-count sentinel). `aggregate('event_study', balance_e=k)` on a bootstrapped fit FAILS CLOSED (`NotImplementedError`: the stored surface is unrestricted and the per-level GMM scores are not retained; re-fit with `n_bootstrap=0`). Analytic fits are unchanged (kit recompute; legacy fields None). The gate is `bootstrap_results is not None`: `None` (n_bootstrap == 0, OR a bootstrapped fit that ended analytical) means the analytical kit RECOMPUTE, the pre-4.0 behaviour. (ii) FAIL-SOFT materialization + storage rule: the surfaces are built inside one guard - any exception leaves them None (the `_es_vcov_ts`/`_es_vcov_index_ts` locals reset too) and the bootstrap runs overall-only (the pre-4.0 plain-fit path); a `_run_bootstrap` attempt that raises (anything but `NotImplementedError`, re-raised) OR RETURNS `None` WITH surfaces present - `_run_bootstrap` returns `None` without an exception when `_compute_cluster_S_scores` hits `_TS_LSMRUnconverged` in the ES or group stage, since the LSMR fallback certifies each RHS column separately and the overall column can succeed while an ES/group column does not - clears them and RETRIES ONCE overall-only (`original_event_study=None, original_group=None, aggregate=None`; `np.random.default_rng(self.seed)` is rebuilt on entry, so a seeded retry reproduces attempt 1's weight draws and the overall quintet equals a plain fit's). A retry failure (a `None` return included) keeps the pre-4.0 catch-all 'Bootstrap failed' warning + analytical overall. STORAGE RULE: the legacy fields hold a surface IFF `bootstrap_results is not None`; when the fit ends analytical they are cleared and `aggregate()` RECOMPUTES analytically. The fail-soft `UserWarning` fires only when the fit DOES carry bootstrap inference and a surface was dropped: 'TwoStageDiD: the event-study/group surfaces could not be materialized on this bootstrapped fit ({e}); the bootstrap covered the overall ATT only, so aggregate('event_study') / aggregate('group') are unavailable here - re-fit with n_bootstrap=0 for the analytical post-fit recompute.'; the relay then raises the shared absent-surface `ValueError` ('This bootstrapped TwoStageDiDResults carries no stored event_study surface: either it was fitted before 4.0 (a bootstrapped plain fit built none) or the surface could not be materialized at fit (see the fit-time warning). Re-fit with diff-diff >= 4.0 for the relayed surface, or with n_bootstrap=0 for the analytical post-fit recompute.'). Net invariant: a bootstrapped plain fit's overall quintet and success/abort outcome are unchanged by the 4.0 removal. (iii) Newly reachable warnings on bootstrapped plain fits: the aggregators' own `UserWarning`s (Stage-1/Stage-2 singular or rank-deficient fallbacks) and the per-target bootstrap-stats `RuntimeWarning`s ('Dropping n/N non-finite bootstrap samples in ... event study (h=...)', 'Too few valid bootstrap samples', 'Bootstrap SE is non-finite or zero'). (iv) Pre-4.0 pickle limitation: a 3.9-3.13 bootstrapped pickle fitted with `fit(aggregate=..., balance_e=k)` stores a balance_e-RESTRICTED surface and records no fit-time `balance_e`, so the relay returns that surface as stored - re-fit under 4.0 for the unrestricted one; one fitted plain stores none and the relay raises the refit `ValueError`. (v) The relay reads the MUTABLE public `event_study_effects`/`group_effects` fields (a view by definition - the StackedDiD precedent); only alpha/anticipation/survey metadata come from kit snapshots, so the recompute route's 'kit snapshots only' invariant does not hold for the relayed dict. (vi) Downstream: `summary()` prints the ES/group tables and `to_dataframe(level='event_study'|'group')` returns frames on bootstrapped plain fits; `DiagnosticReport`/`BusinessReport` run `heterogeneity` (on the relayed GROUP percentile dict - `_collect_effect_scalars` reads `group_effects` first) and `parallel_trends` (`pretrends=True`) on the relayed surface instead of skipping with `NotImplementedError` (no sensitivity check applies); `plot_event_study(results)` plots them natively. Replicate-weight designs never meet the always-on surfaces (replicate + `n_bootstrap > 0` is rejected before them), so the fit-time replicate stack is always `[overall]`-only - the degenerate-design overall-row migration delta in (e) stands. - **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- diff --git a/docs/methodology/REPORTING.md b/docs/methodology/REPORTING.md index 018fe64b..ab98413b 100644 --- a/docs/methodology/REPORTING.md +++ b/docs/methodology/REPORTING.md @@ -23,8 +23,9 @@ existing diff-diff utility (`compute_honest_did`, `HonestDiD.sensitivity`, `BaconDecomposition`, `check_parallel_trends`, `compute_pretrends_power`). The post-fit derivation is the one report-layer data source beyond direct reads: -when a fit's raw `event_study_effects` field is absent (the modern, -no-fit-time-`aggregate=` path), `DiagnosticReport` resolves +when a fit's raw `event_study_effects` field is absent (every analytic +CS/DMLDiD/EfficientDiD/ImputationDiD/TwoStageDiD/ContinuousDiD fit since +the 4.0 removal of the fit-time `aggregate=` keyword), `DiagnosticReport` resolves `results.aggregate('event_study')` once (cached) and consumes the returned `EventStudyResults` container, where applicable per estimator: CallawaySantAnna and DMLDiD for parallel_trends / @@ -48,14 +49,20 @@ joint-Wald path and pretrends_power/sensitivity the full covariance - the Bonferroni and diagonal fallbacks remain for fits where no bootstrap covariance is stored, e.g. `n_bootstrap` at or below the number of event times - and the replay's re-emitted fit-time warnings -are recorded and republished per section). Kit-less/legacy-pickle fits, -backend-mismatched CS bootstrap replays, and the other estimators' -bootstrap gates still fail closed: the derivation exception is caught -and surfaced as an explicit per-check skip reason, never substituted -with analytical numbers. The raw field, when present — including the -requested-but-empty `{}` sentinel, which encodes fit-time -configuration such as a `balance_e=` that emptied the window — always -takes precedence and is never re-derived. When the caller +are recorded and republished per section). Bootstrapped ImputationDiD / +TwoStageDiD / ContinuousDiD fits are HYBRIDS since 4.0 (PR 3c): they +materialize the percentile event-study (and group) surface at fit into +the raw fields, so the report reads those directly (raw-field +precedence) rather than deriving. Only kit-less/legacy-pickle fits, +backend-mismatched CS bootstrap replays, and bootstrapped hybrid fits +whose surface could not be materialized (fail-soft cleared, the relay +raises) still skip: the derivation exception is caught and surfaced as +an explicit per-check skip reason, never substituted with analytical +numbers. The raw field, when present — including the +requested-but-empty `{}` sentinel, which now arises only from producers +that materialize at fit (StackedDiD, SunAbraham, the bootstrapped +hybrids; always `balance_e=None`) — always takes precedence and is +never re-derived. When the caller passes the raw panel + column kwargs, `DiagnosticReport` may call those utilities on the supplied data (2x2 PT via `check_parallel_trends`, Goodman-Bacon decomposition via diff --git a/docs/methodology/variance-conventions.md b/docs/methodology/variance-conventions.md index 1f90769a..6f38beaa 100644 --- a/docs/methodology/variance-conventions.md +++ b/docs/methodology/variance-conventions.md @@ -143,9 +143,10 @@ output). characterization-anchored only (DoubleML's RCS score differs and omits the Chang lambda-correction; equation-level fixtures validate the variance). **ImputationDiD is conditional**: its pretrends=True lead regression DOES - run the shared clustered CR1 — surfaced fit-time via the deprecated - aggregate="event_study" or post-fit via results.aggregate('event_study') - since the M-021 migration — and carries the K_reference increment there + run the shared clustered CR1 — surfaced via results.aggregate('event_study') + since the M-021 migration (analytic fits recompute it; bootstrapped fits + relay the surface materialized at fit, 4.0 PR 3c) — and carries the + K_reference increment there (+6, the [time, unit] no-intercept increment on df_0 — pinned via expected_adjustment on its matrix row). - **L4 — hc2/hc2_bm/hc3** (leverage / Satterthwaite DOF — no CR1 factor; hc3 squares the leverage denominator and is one-way only; under `fweight` the leverage is each replicate row's UNWEIGHTED quadratic form against the weighted bread — frequency weights are replicated data, and compressed HC2/HC3 equal literal expansion exactly — while aweight/pweight keep the WLS-hat `w_i x_i'(X'WX)^{-1}x_i` convention), diff --git a/docs/migration-4.0.md b/docs/migration-4.0.md index 74ddbb12..d443869d 100644 --- a/docs/migration-4.0.md +++ b/docs/migration-4.0.md @@ -133,9 +133,10 @@ may still need a replacement call — see its `Fix` cell. **Dropped `fit()` parameters shift the positional slots after them.** Removing the fit-time `aggregate=`/`balance_e=` keywords moves every later positional parameter left: on -`CallawaySantAnna` `survey_design` moves from slot 9 to 7; on `EfficientDiD` `survey_design` -9 -> 7 and `store_eif` 10 -> 8; on `StackedDiD` (`aggregate` sat before them) -`population`/`survey_design`/`covariates` move 7/8/9 -> 6/7/8; on `ChaisemartinDHaultfoeuille` +`CallawaySantAnna`, `ImputationDiD` and `TwoStageDiD` `survey_design` moves from slot 9 to 7; +on `EfficientDiD` `survey_design` 9 -> 7 and `store_eif` 10 -> 8; on `StackedDiD` (`aggregate` +sat before them) `population`/`survey_design`/`covariates` move 7/8/9 -> 6/7/8; on +`ContinuousDiD` `survey_design`/`covariates` move 8/9 -> 7/8; on `ChaisemartinDHaultfoeuille` `L_max`, `covariates`, `trends_linear`, `trends_nonparam`, `honest_did`, `heterogeneity`, `design2` and `survey_design` each move one slot left (7..14 -> 6..13). `HeterogeneousAdoptionDiD` and `did_had_pretest_workflow` shift nothing (everything after the removed slot is keyword-only; @@ -167,18 +168,27 @@ surface (nothing to derive), and ChaisemartinDHaultfoeuille's pre-period checks read `placebo_event_study` directly. ```{warning} -**Bootstrapped fits: CallawaySantAnna, DMLDiD, and EfficientDiD are covered; the remaining recompute adopters are not yet.** -On `CallawaySantAnna`, `DMLDiD` (which inherits the CS replay channel), and `EfficientDiD`, the post-fit recompute levels now REPLAY the +**Bootstrapped fits: every adopter has a post-fit route - two mechanisms.** +On `CallawaySantAnna`, `DMLDiD` (which inherits the CS replay channel), and `EfficientDiD`, the post-fit recompute levels REPLAY the fit-time multiplier bootstrap from the fit-retained RNG state (percentile inference -matching a fit-time aggregation to floating-point reassociation) — no fit-time keyword -needed; only pre-replay legacy pickles and artifacts moved across the Rust/NumPy weight -backend fail closed with a refit message. On `ImputationDiD`, `TwoStageDiD` and -`ContinuousDiD`, the recompute levels still raise `NotImplementedError` when the fit used -`n_bootstrap > 0` — keep the fit-time call there for now and track their open `TODO.md` -rows. -`aggregate("simple")` — and, on its five adopters, `aggregate("total")` (3.10; DMLDiD joined in 3.11 — panel non-survey fits only: RCS fits, declared-`survey_design=` fits, and bare-`cluster=` fits with incomplete treated support fail `total` closed) — does relay, -and `StackedDiD`, `ChaisemartinDHaultfoeuille` and `HeterogeneousAdoptionDiD` are unaffected — -their `aggregate()` is a pure view over stored fields. +matching a fit-time aggregation to floating-point reassociation); only pre-replay legacy +pickles and artifacts moved across the Rust/NumPy weight backend fail closed with a refit +message. On `ImputationDiD`, `TwoStageDiD` and `ContinuousDiD` a fit with `n_bootstrap > 0` +ALWAYS computes the event-study (and, on Imputation/TwoStage, group) percentile surface at fit +and `aggregate('event_study')` / `aggregate('group')` RELAYS it - the same numbers the +3.x `fit(aggregate="all")` route produced, with a NaN `df` column and no `vcov`. Two limits +there: `balance_e=` is unavailable on those bootstrapped fits (`NotImplementedError` - the +stored surface was computed unrestricted; re-fit with `n_bootstrap=0` for a balanced window), +and a bootstrapped pickle written by 3.9-3.13 stores no surface (or, if it was fitted with +`balance_e=`, a restricted one), so re-fit under 4.0 for the relayed surface. If the surface +could not be materialized at fit, the fit warns once, the headline bootstrap is unchanged, +and the relay raises with that reason. +`aggregate("simple")` - and, on its five adopters, `aggregate("total")` (3.10; DMLDiD joined in 3.11 - panel non-survey fits only: RCS fits, declared-`survey_design=` fits, and bare-`cluster=` fits with incomplete treated support fail `total` closed) - relays on every adopter, +and `StackedDiD`, `ChaisemartinDHaultfoeuille` and `HeterogeneousAdoptionDiD` are unaffected - +their `aggregate()` is a pure view over stored fields. One residual delta on +replicate-weight `ImputationDiD`/`TwoStageDiD` designs: the plain fit's replicate stack is +`[overall]`-only, so on degenerate designs the OVERALL row's se/CI/df can differ from what a +3.x `fit(aggregate=...)` call reported (each surface self-consistent). ``` ## Results fields @@ -387,16 +397,16 @@ does not mean no action is required, so read the `Fix` cell. | Row | Group | Old | New | Fix | |---|---|---|---|---| | M-020 | aggregate-postfit | `diff_diff:CallawaySantAnna.fit[aggregate]` | `diff_diff:CallawaySantAnnaResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits replay the fit-time bootstrap post-fit - see the aggregation section. | -| M-021 | aggregate-postfit | `diff_diff:ImputationDiD.fit[aggregate]` | `diff_diff:ImputationDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. On a bootstrapped fit the recompute levels raise today - see the aggregation section. | -| M-022 | aggregate-postfit | `diff_diff:TwoStageDiD.fit[aggregate]` | `diff_diff:TwoStageDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. On a bootstrapped fit the recompute levels raise today - see the aggregation section. | +| M-021 | aggregate-postfit | `diff_diff:ImputationDiD.fit[aggregate]` | `diff_diff:ImputationDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits compute the surface at fit and `aggregate()` relays it - see the aggregation section. | +| M-022 | aggregate-postfit | `diff_diff:TwoStageDiD.fit[aggregate]` | `diff_diff:TwoStageDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits compute the surface at fit and `aggregate()` relays it - see the aggregation section. | | M-023 | aggregate-postfit | `diff_diff:EfficientDiD.fit[aggregate]` | `diff_diff:EfficientDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits replay the fit-time bootstrap post-fit - see the aggregation section. | | M-024 | aggregate-postfit | `diff_diff:StackedDiD.fit[aggregate]` | `diff_diff:StackedDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. | -| M-025 | aggregate-postfit | `diff_diff:ContinuousDiD.fit[aggregate]` | `diff_diff:ContinuousDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. On a bootstrapped fit the recompute levels raise today - see the aggregation section. | +| M-025 | aggregate-postfit | `diff_diff:ContinuousDiD.fit[aggregate]` | `diff_diff:ContinuousDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits compute the surface at fit and `aggregate()` relays it - see the aggregation section. | | M-026 | aggregate-postfit | `diff_diff:ChaisemartinDHaultfoeuille.fit[aggregate]` | `diff_diff:ChaisemartinDHaultfoeuilleResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. | | M-027 | aggregate-postfit | `diff_diff:HeterogeneousAdoptionDiD.fit[aggregate]` | `diff_diff:HeterogeneousAdoptionDiDResults.aggregate` | Move `aggregate=` off `fit()` onto post-fit `results.aggregate(...)`. | | M-117 | aggregate-postfit | `diff_diff:CallawaySantAnna.fit[balance_e]` | `diff_diff:CallawaySantAnnaResults.aggregate[balance_e]` | Move `balance_e=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits replay the fit-time bootstrap post-fit - see the aggregation section. | -| M-118 | aggregate-postfit | `diff_diff:ImputationDiD.fit[balance_e]` | `diff_diff:ImputationDiDResults.aggregate[balance_e]` | Move `balance_e=` off `fit()` onto post-fit `results.aggregate(...)`. On a bootstrapped fit the recompute levels raise today - see the aggregation section. | -| M-119 | aggregate-postfit | `diff_diff:TwoStageDiD.fit[balance_e]` | `diff_diff:TwoStageDiDResults.aggregate[balance_e]` | Move `balance_e=` off `fit()` onto post-fit `results.aggregate(...)`. On a bootstrapped fit the recompute levels raise today - see the aggregation section. | +| M-118 | aggregate-postfit | `diff_diff:ImputationDiD.fit[balance_e]` | `diff_diff:ImputationDiDResults.aggregate[balance_e]` | Move `balance_e=` off `fit()` onto post-fit `results.aggregate(...)`. Unavailable on a bootstrapped fit (`NotImplementedError`; re-fit with `n_bootstrap=0`) - see the aggregation section. | +| M-119 | aggregate-postfit | `diff_diff:TwoStageDiD.fit[balance_e]` | `diff_diff:TwoStageDiDResults.aggregate[balance_e]` | Move `balance_e=` off `fit()` onto post-fit `results.aggregate(...)`. Unavailable on a bootstrapped fit (`NotImplementedError`; re-fit with `n_bootstrap=0`) - see the aggregation section. | | M-120 | aggregate-postfit | `diff_diff:EfficientDiD.fit[balance_e]` | `diff_diff:EfficientDiDResults.aggregate[balance_e]` | Move `balance_e=` off `fit()` onto post-fit `results.aggregate(...)`. Bootstrapped fits replay the fit-time bootstrap post-fit - see the aggregation section. | | M-139 | aggregate-postfit | `diff_diff:did_had_pretest_workflow[aggregate]` | — | Remove `aggregate=`; the battery is inferred from panel shape - two periods select the overall battery, more than two select the event-study battery. It is not selected post-fit (`HADPretestReport.aggregate` is a metadata field, not a method). | | M-060 | alias-table | `EventStudy` | — | Import `TwoWayFixedEffects(...).fit(..., event_study=True)` - the alias is dropped, not retargeted instead of the `EventStudy` alias. | diff --git a/docs/performance-plan.md b/docs/performance-plan.md index 4d885624..109bbb05 100644 --- a/docs/performance-plan.md +++ b/docs/performance-plan.md @@ -835,11 +835,12 @@ These are developer-ergonomics / API-consistency smells surfaced during scenario development. None are silent-failures and none belong in this PR or in the silent-failures audit; logging here for awareness. -1. **`aggregate` / `level` parameter naming is inconsistent.** CS accepts - `aggregate="event_study"`; ContinuousDiD historically required +1. **`aggregate` / `level` parameter naming was inconsistent (historical; resolved at + 4.0).** CS accepted `aggregate="event_study"` on `fit()`; ContinuousDiD required `aggregate="eventstudy"` on `fit()` **but** `level="event_study"` on `to_dataframe()`. Two different spellings within one estimator plus a - third cross-estimator spelling. Surfaced when the P1 exit-propagation + third cross-estimator spelling. Both fit-time keywords were removed at 4.0 (PRs 3b/3c); + `results.aggregate("event_study")` is the single post-fit spelling. Surfaced when the P1 exit-propagation fix stopped silently swallowing the resulting `ValueError` in the dose-response benchmark. RESOLVED by row M-025: the fit-time kwarg is deprecated (removed in 4.0, its `"eventstudy"` spelling with it) and diff --git a/docs/performance-scenarios.md b/docs/performance-scenarios.md index 8f46cb13..90399120 100644 --- a/docs/performance-scenarios.md +++ b/docs/performance-scenarios.md @@ -307,7 +307,7 @@ serves a different purpose: R-parity accuracy). They complement it. ``` - **Operation chain.** (1) ContinuousDiD fit - produces overall ATT, overall ACRT, and the dose-response curves - unconditionally (the fit-time `aggregate=` kwarg is deprecated, row + unconditionally (`fit()` takes no `aggregate=` kwarg since 4.0, row M-025); (2) extract `results.to_dataframe(level="dose_response")` and `level="group_time"`; (3) a second analytical ContinuousDiD fit followed by diff --git a/docs/tutorials/14_continuous_did.ipynb b/docs/tutorials/14_continuous_did.ipynb index 8f1a5219..1f79ae3f 100644 --- a/docs/tutorials/14_continuous_did.ipynb +++ b/docs/tutorials/14_continuous_did.ipynb @@ -177,7 +177,7 @@ "source": [ "## Basic Estimation\n", "\n", - "The `ContinuousDiD` estimator follows an sklearn-like API. Call `fit()` with column names for outcome, unit, time, first treatment period, and dose. The dose-response curve and global summary parameters are always computed (the fit-time `aggregate=` keyword is deprecated; aggregation is a post-fit step via `results.aggregate(...)`)." + "The `ContinuousDiD` estimator follows an sklearn-like API. Call `fit()` with column names for outcome, unit, time, first treatment period, and dose. The dose-response curve and global summary parameters are always computed; aggregation is a post-fit step via `results.aggregate(...)` (`fit()` takes no `aggregate=` keyword since 4.0)." ] }, { @@ -317,7 +317,7 @@ "\n", "Event study works best with multiple cohorts (to have richer variation in event-time). A single cohort can still produce event-time estimates, but diagnostics are more informative with multiple cohorts. We generate a new dataset with two treatment cohorts.\n", "\n", - "The event study is a **post-fit** aggregation: call `results.aggregate(\"event_study\")` on an analytical fit (the fit-time `aggregate=\"eventstudy\"` keyword is deprecated). On a bootstrapped fit the post-fit event study raises - re-fit with `n_bootstrap=0`, or keep the deprecated fit-time keyword until 4.0 if you need bootstrap event-study inference." + "The event study is a **post-fit** aggregation: call `results.aggregate(\"event_study\")`. On an analytical fit it recomputes the binarized event study from the retained kit; on a bootstrapped fit (`n_bootstrap > 0`) the percentile event study is computed at fit and `aggregate(\"event_study\")` relays it (4.0). The 3.x fit-time `aggregate=\"eventstudy\"` keyword was removed at 4.0." ] }, { @@ -646,7 +646,7 @@ "\n", "- **`level=\"dose_response\"`** (default): The dose-response curve with ATT(d) and ACRT(d)\n", "- **`level=\"group_time\"`**: Underlying group-time cell estimates\n", - "- **`level=\"event_study\"`**: Event study effects (legacy surface - populated only by the deprecated fit-time `aggregate=\"eventstudy\"`; prefer the post-fit `results.aggregate(\"event_study\")` container and its `to_dataframe()`)" + "- **`level=\"event_study\"`**: Event study effects (legacy surface - populated only on bootstrapped fits since 4.0; prefer the post-fit `results.aggregate(\"event_study\")` container and its `to_dataframe()`)" ] }, { @@ -697,7 +697,7 @@ "| `n_bootstrap` | 0 | Bootstrap iterations (0 = analytical SEs only) |\n", "| `bootstrap_weights` | `\"rademacher\"` | `\"rademacher\"`, `\"mammen\"`, or `\"webb\"` |\n", "| `seed` | `None` | Random seed for reproducibility |\n", - "| `aggregate` (fit) | deprecated | Fit-time aggregation is deprecated (row M-025); use post-fit `results.aggregate(\"event_study\"/\"dose\"/\"simple\")` - the dose curves are always computed |\n", + "| `aggregate` (fit) | removed at 4.0 | Fit-time aggregation was removed (row M-025); use post-fit `results.aggregate(\"event_study\"/\"dose\"/\"simple\")` - the dose curves are always computed |\n", "\n", "### Reference\n", "\n", diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index dc6318b9..39dda237 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -251,12 +251,12 @@ rows: introduced_in: "3.9" deprecated_in: "3.9" removed_in: "4.0" - status: shimmed + status: removed phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py code_refs: [diff_diff/imputation.py, diff_diff/imputation_aggregation.py, diff_diff/imputation_results.py, diff_diff/imputation_bootstrap.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] - notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-118], warn-and-still-work; the imputation_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-070]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary - a behavior improvement). The successor is a PANEL-BACKED lazy recompute kit: ES/group aggregation is a target-specific Theorem-3 recompute from the working panel + untreated FE model (no compact influence payload can honor a different balance_e), so the kit's bookkeeping holds REFERENCES to the SAME per-fit objects self._fit_data already retains for pretrend_test() - ZERO marginal memory and unchanged pickles via memoization (the _estimator_ref field already ships the panel; enumeration in the REGISTRY ImputationDiD Note). Value snapshots (treatment_groups copy, config scalars, a dataclasses.replace copy of survey_metadata, and the survey_df_seed/survey_df_final df channels) isolate recompute and the ES carrier from public-field mutation; aggregate() reads NOTHING mutable off the results object except the deliberate overall-quintet relay and the bootstrap_results gate. SUPPORTED SUBSET simple|event_study|group|total ('total' since 2026-08-16: exact relay C x overall with C = the FINITE-tau complete-case support snapshot 'total_support', stashed at kit build alongside the panel-backed refs BECAUSE the kit's df is a live _fit_data reference - the one non-ref value snapshot this row's zero-marginal-memory claim gains; pickles still unchanged via memoization since the stash is a float; declared-survey_design fits and pre-upgrade kits missing the stash raise NotImplementedError); calendar/'all' fail closed via the mixin; weights= rejected. Bootstrap fits: 'simple' and 'total' RELAY the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (the per-target psi machinery makes replay tractable - a TODO row). The prior fail-closed-for-ALL-levels uniform-parity decision was superseded 2026-08-05 with the [M-027] per-level convergence; its rationale - never publish analytical provenance beside percentile inference - is honored by the NaN df column. Replicate-weight fits REPLAY the extracted _replicate_override_aggregates with a LEVEL-MATCHED stack ([overall, ES] or [overall, groups]): compute_replicate_refit_variance validates replicates jointly, so aggregate(L) reproduces fit(aggregate=L) exactly and a fit(aggregate='all') surface is NOT the equivalence target when a replicate NaNs on one family's targets; the same joint-stack coupling means migrating a replicate fit from fit(aggregate=) to plain fit changes the OVERALL row's se/CI/df on degenerate designs (documented migration delta, CHANGELOG + REGISTRY note (e) + a contract-test pin). pretrends=True + replicate: post-fit aggregate('event_study') raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); group/simple still work. Simple relay: n = n_treated_obs with n_kind='obs' (the treated/control unit sets OVERLAP, so the CS/EDiD disjoint-units convention cannot apply - the StackedDiD carve-out class); df = the survey_df_final snapshot (what the stored overall inference received). Group rows record per-row df_used at each safe_inference call (additive row-dict key; the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via .get). The M-127 df_convention inert-config warning predicate is REVISED to reachability (pretrends AND not-replicate AND (deprecated fit-time ES/all supplied OR n_bootstrap <= 0)) because post-fit aggregate() made the old aggregate-keyed claim false; reachability-BASED, not exact - a fit whose bootstrap later fails (bootstrap_results=None) can still aggregate post-fit, so that corner warns spuriously (recorded on M-127 too). Container admission NOT widened: ImputationDiD is rejected BY DESIGN (no joint ES covariance - per-horizon conservative SEs only; see M-093). balance_e moves as its own row [M-118]. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." + notes: "REMOVED at the 4.0 cut (PR 3c, 2026-10-21). Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-118], warn-and-still-work; the imputation_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-070]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary - a behavior improvement). The successor is a PANEL-BACKED lazy recompute kit: ES/group aggregation is a target-specific Theorem-3 recompute from the working panel + untreated FE model (no compact influence payload can honor a different balance_e), so the kit's bookkeeping holds REFERENCES to the SAME per-fit objects self._fit_data already retains for pretrend_test() - ZERO marginal memory and unchanged pickles via memoization (the _estimator_ref field already ships the panel; enumeration in the REGISTRY ImputationDiD Note). Value snapshots (treatment_groups copy, config scalars, a dataclasses.replace copy of survey_metadata, and the survey_df_seed/survey_df_final df channels) isolate recompute and the ES carrier from public-field mutation; aggregate() reads NOTHING mutable off the results object except the deliberate overall-quintet relay and the bootstrap_results gate. SUPPORTED SUBSET simple|event_study|group|total ('total' since 2026-08-16: exact relay C x overall with C = the FINITE-tau complete-case support snapshot 'total_support', stashed at kit build alongside the panel-backed refs BECAUSE the kit's df is a live _fit_data reference - the one non-ref value snapshot this row's zero-marginal-memory claim gains; pickles still unchanged via memoization since the stash is a float; declared-survey_design fits and pre-upgrade kits missing the stash raise NotImplementedError); calendar/'all' fail closed via the mixin; weights= rejected. Bootstrap fits: 'simple' and 'total' RELAY the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (the per-target psi machinery makes replay tractable - a TODO row). The prior fail-closed-for-ALL-levels uniform-parity decision was superseded 2026-08-05 with the [M-027] per-level convergence; its rationale - never publish analytical provenance beside percentile inference - is honored by the NaN df column. Replicate-weight fits REPLAY the extracted _replicate_override_aggregates with a LEVEL-MATCHED stack ([overall, ES] or [overall, groups]): compute_replicate_refit_variance validates replicates jointly, so aggregate(L) reproduces fit(aggregate=L) exactly and a fit(aggregate='all') surface is NOT the equivalence target when a replicate NaNs on one family's targets; the same joint-stack coupling means migrating a replicate fit from fit(aggregate=) to plain fit changes the OVERALL row's se/CI/df on degenerate designs (documented migration delta, CHANGELOG + REGISTRY note (e) + a contract-test pin). pretrends=True + replicate: post-fit aggregate('event_study') raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); group/simple still work. Simple relay: n = n_treated_obs with n_kind='obs' (the treated/control unit sets OVERLAP, so the CS/EDiD disjoint-units convention cannot apply - the StackedDiD carve-out class); df = the survey_df_final snapshot (what the stored overall inference received). Group rows record per-row df_used at each safe_inference call (additive row-dict key; the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via .get). The M-127 df_convention inert-config warning predicate is REVISED to reachability (pretrends AND not-replicate AND (deprecated fit-time ES/all supplied OR n_bootstrap <= 0)) because post-fit aggregate() made the old aggregate-keyed claim false; reachability-BASED, not exact - a fit whose bootstrap later fails (bootstrap_results=None) can still aggregate post-fit, so that corner warns spuriously (recorded on M-127 too). Container admission NOT widened: ImputationDiD is rejected BY DESIGN (no joint ES covariance - per-horizon conservative SEs only; see M-093). balance_e moves as its own row [M-118]. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons). AT 4.0 (PR 3c): fits with n_bootstrap > 0 ALWAYS compute the event-study AND group surfaces at fit (balance_e=None - exactly what fit(aggregate='all') computed), store them in the legacy event_study_effects/group_effects fields, and aggregate('event_study'|'group') RELAYS the stored surface (kit-snapshot alpha/anticipation/survey metadata; NaN df column, vcov None) - the recompute levels no longer fail closed on bootstrapped fits; balance_e is not None on a bootstrapped fit raises NotImplementedError [M-118]. Analytic fits are unchanged (legacy fields None, kit recompute on demand). STORAGE RULE: the legacy fields hold a surface IFF the fit carries bootstrap inference (bootstrap_results is not None); a bootstrapped fit that ends analytical (n_valid == 0, total bootstrap failure) clears them and aggregate() recomputes analytically; a fit WITH bootstrap inference and no stored surface (pre-4.0 pickle, fail-soft drop) raises the shared absent-surface ValueError. Materialization is FAIL-SOFT with a one-shot overall-only retry (any exception while building the surfaces, or a bootstrap attempt that raises/returns None with surfaces present, leaves them None, warns once 'could not be materialized', and re-runs the bootstrap overall-only - on a seeded fit the retry reproduces attempt 1's weight draws), so the headline bootstrap quintet and the fit's success/abort outcome are unchanged by the removal. The pretrends+replicate fit-time gate was deleted (unreachable: replicate + bootstrap already raises earlier; the analytic combination is caught post-fit); the df_convention reachability predicate is now pretrends AND not replicate [M-127]. The aggregators' own UserWarnings and the per-target bootstrap-stats RuntimeWarnings now fire on bootstrapped plain fits; summary() prints and to_dataframe(level=) returns the ES/group tables there; DiagnosticReport/BusinessReport run on the relayed surface. Known limitation: a 3.9-3.13 bootstrapped pickle fitted with balance_e=k stores a balance_e-RESTRICTED surface and the relay returns it as stored. Positional slot shift: survey_design 9 -> 7." - id: M-022 kind: param group: aggregate-postfit @@ -265,12 +265,12 @@ rows: introduced_in: "3.9" deprecated_in: "3.9" removed_in: "4.0" - status: shimmed + status: removed phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py code_refs: [diff_diff/two_stage.py, diff_diff/two_stage_aggregation.py, diff_diff/two_stage_results.py, diff_diff/two_stage_bootstrap.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] - notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-119], warn-and-still-work; the two_stage_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-071]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary). The successor is a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich, so the kit retains a COLUMN-SUBSET COPY of the working frame (only the columns the moved methods read by name, deduplicated - cluster= may legally name the unit/time/first_treat column) plus the Stage-1 FE model, masks, and survey objects. MEMORY CONTRACT: this is the FIRST panel retention on TwoStageDiD results - O(n_obs) incl. unit/time/cluster identifier columns on every results object and pickle, and replicate designs additionally retain the (n_obs x R) replicate matrix via resolved_survey; the CS/EDiD identifier-minimization guarantee deliberately does NOT hold (a store_kit opt-out is a DEFERRED row). score_pad_mask/cluster_ids_full are stored as the Wave-E.3-GATED values fit actually passed. Value snapshots (treatment_groups copy, overall_att, survey_df_stage2/survey_df_final, a dataclasses.replace copy of survey_metadata) isolate recompute and the ES carrier from public-field mutation. SUPPORTED SUBSET simple|event_study|group|total ('total' since 2026-08-16: exact relay C x overall with C = the post-filter treatment-indicator support - omega_1_mask minus non-finite _y_tilde rows, read from the kit's private column-subset frame COPY, so no fit-time stash is needed; declared-survey_design fits raise NotImplementedError); calendar/'all' fail closed; weights= rejected. Bootstrap fits: 'simple' and 'total' RELAY the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform fail-closed decision superseded 2026-08-05 with the [M-027] per-level convergence; a fit whose bootstrap FAILED (bootstrap_results=None, analytical inference retained) aggregates normally. Replicate-weight fits REPLAY the extracted _replay_replicate_inference with a LEVEL-MATCHED stack - aggregate(L) reproduces fit(aggregate=L) exactly; fit(aggregate='all') is NOT the equivalence target on degenerate designs, and the same joint-stack coupling makes the migration to plain fit change the OVERALL row's se/CI/df there (documented migration delta). Post-fit aggregate('event_study') reproduces the M-092 container contract exactly: analytical fits thread the recomputed joint vcov + vcov_index + the finite-and->0 df scalar through the carrier; replicate fits thread vcov=None/index=None with the REPLAYED level-matched df. Simple relay: n = n_treated_obs with n_kind='obs' (overlapping unit sets - StackedDiD carve-out class); df = the survey_df_final snapshot (on replicate fits that value came from the [overall]-only stack - snapshotted, never re-derived). Group relay df is a SCALAR broadcast (deliberate divergence from ImputationDiD's per-row df_used: _stage2_group passes one immutable survey_df to every row, so the scalar is provenance-exact by construction and the moved method stays verbatim). Container admission NOT widened: DEFERRED pending a normalization derivation - analytical surfaces carry the real joint Gardner-GMM covariance (M-092), but pre-period coefficients are stage-1 residual means, not reference-normalized contrasts, while HonestDiD's Delta arithmetic hard-codes delta_0=0 (see M-093 + the DEFERRED.md paper-gated row). balance_e moves as its own row [M-119]. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." + notes: "REMOVED at the 4.0 cut (PR 3c, 2026-10-21). Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (plain fit() never warns; supplying ANY value, None included, warns - CS-style joint warning with balance_e [M-119], warn-and-still-work; the two_stage_did wrapper forwards the sentinel so a plain wrapper call never fires the aggregate warning; since 3.9 the wrapper itself warns per [M-071]). NO fit-time value validation existed and none is added (unknown strings silently act like None; the post-fit successor fails closed via the mixin vocabulary). The successor is a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich, so the kit retains a COLUMN-SUBSET COPY of the working frame (only the columns the moved methods read by name, deduplicated - cluster= may legally name the unit/time/first_treat column) plus the Stage-1 FE model, masks, and survey objects. MEMORY CONTRACT: this is the FIRST panel retention on TwoStageDiD results - O(n_obs) incl. unit/time/cluster identifier columns on every results object and pickle, and replicate designs additionally retain the (n_obs x R) replicate matrix via resolved_survey; the CS/EDiD identifier-minimization guarantee deliberately does NOT hold (a store_kit opt-out is a DEFERRED row). score_pad_mask/cluster_ids_full are stored as the Wave-E.3-GATED values fit actually passed. Value snapshots (treatment_groups copy, overall_att, survey_df_stage2/survey_df_final, a dataclasses.replace copy of survey_metadata) isolate recompute and the ES carrier from public-field mutation. SUPPORTED SUBSET simple|event_study|group|total ('total' since 2026-08-16: exact relay C x overall with C = the post-filter treatment-indicator support - omega_1_mask minus non-finite _y_tilde rows, read from the kit's private column-subset frame COPY, so no fit-time stash is needed; declared-survey_design fits raise NotImplementedError); calendar/'all' fail closed; weights= rejected. Bootstrap fits: 'simple' and 'total' RELAY the stored percentile quintet verbatim (finite safe_inference t included) with a NaN df column, while the recompute levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform fail-closed decision superseded 2026-08-05 with the [M-027] per-level convergence; a fit whose bootstrap FAILED (bootstrap_results=None, analytical inference retained) aggregates normally. Replicate-weight fits REPLAY the extracted _replay_replicate_inference with a LEVEL-MATCHED stack - aggregate(L) reproduces fit(aggregate=L) exactly; fit(aggregate='all') is NOT the equivalence target on degenerate designs, and the same joint-stack coupling makes the migration to plain fit change the OVERALL row's se/CI/df there (documented migration delta). Post-fit aggregate('event_study') reproduces the M-092 container contract exactly: analytical fits thread the recomputed joint vcov + vcov_index + the finite-and->0 df scalar through the carrier; replicate fits thread vcov=None/index=None with the REPLAYED level-matched df. Simple relay: n = n_treated_obs with n_kind='obs' (overlapping unit sets - StackedDiD carve-out class); df = the survey_df_final snapshot (on replicate fits that value came from the [overall]-only stack - snapshotted, never re-derived). Group relay df is a SCALAR broadcast (deliberate divergence from ImputationDiD's per-row df_used: _stage2_group passes one immutable survey_df to every row, so the scalar is provenance-exact by construction and the moved method stays verbatim). Container admission NOT widened: DEFERRED pending a normalization derivation - analytical surfaces carry the real joint Gardner-GMM covariance (M-092), but pre-period coefficients are stage-1 residual means, not reference-normalized contrasts, while HonestDiD's Delta arithmetic hard-codes delta_0=0 (see M-093 + the DEFERRED.md paper-gated row). balance_e moves as its own row [M-119]. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons). AT 4.0 (PR 3c): fits with n_bootstrap > 0 ALWAYS compute the event-study AND group surfaces at fit (balance_e=None - exactly what fit(aggregate='all') computed), store them in the legacy event_study_effects/group_effects fields, and aggregate('event_study'|'group') RELAYS the stored surface (kit-snapshot alpha/anticipation/survey metadata; NaN df column, vcov None) - the recompute levels no longer fail closed on bootstrapped fits; balance_e is not None on a bootstrapped fit raises NotImplementedError [M-119]. Analytic fits are unchanged (legacy fields None, kit recompute on demand). STORAGE RULE: the legacy fields hold a surface IFF the fit carries bootstrap inference (bootstrap_results is not None); a bootstrapped fit that ends analytical (n_valid == 0, total bootstrap failure) clears them and aggregate() recomputes analytically; a fit WITH bootstrap inference and no stored surface (pre-4.0 pickle, fail-soft drop) raises the shared absent-surface ValueError. Materialization is FAIL-SOFT with a one-shot overall-only retry (any exception while building the surfaces, or a bootstrap attempt that raises/returns None with surfaces present, leaves them None, warns once 'could not be materialized', and re-runs the bootstrap overall-only - on a seeded fit the retry reproduces attempt 1's weight draws), so the headline bootstrap quintet and the fit's success/abort outcome are unchanged by the removal. The aggregators' own UserWarnings and the per-target bootstrap-stats RuntimeWarnings now fire on bootstrapped plain fits; summary() prints and to_dataframe(level=) returns the ES/group tables there; DiagnosticReport/BusinessReport run on the relayed surface. Known limitation: a 3.9-3.13 bootstrapped pickle fitted with balance_e=k stores a balance_e-RESTRICTED surface and the relay returns it as stored. Positional slot shift: survey_design 9 -> 7." - id: M-023 kind: param group: aggregate-postfit @@ -307,12 +307,12 @@ rows: introduced_in: "3.9" deprecated_in: "3.9" removed_in: "4.0" - status: shimmed + status: removed phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py code_refs: [diff_diff/continuous_did.py, diff_diff/continuous_did_aggregation.py, diff_diff/continuous_did_results.py, diff_diff/aggregation.py, diff_diff/results_base.py, diff_diff/honest_did.py, diff_diff/pretrends.py, diff_diff/practitioner.py, diff_diff/guides/llms-practitioner.txt, diff_diff/diagnostic_report.py] - notes: "Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (a plain fit() never warns; supplying ANY value incl. None warns once, then the legacy routing runs unchanged). Unlike the EfficientDiD/Imputation shims, fit-time VALUE VALIDATION pre-existed and STAYS: unknown strings still raise ValueError after the warning ((None, 'dose', 'eventstudy') only). The no-underscore 'eventstudy' spelling dies with the param in 4.0; aggregate() accepts only the unified vocabulary + 'dose' as this estimator's documented extra level. MIXED VIEW/RECOMPUTE architecture (unique in the register): 'simple' and 'dose' are pure VIEWS over stored public fields - the dose curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by fit (aggregate='dose' was a fit-time no-op) - so both levels are PERMITTED on bootstrap fits (the library-wide per-level relay rule, since [M-027] converged CS/EDiD/Imputation/TwoStage onto it), relaying stored inference verbatim: the overall rows carry the FINITE safe_inference t fit stores beside percentile p/CI, the dose rows reproduce DoseResponseCurve.to_dataframe (NaN t under bootstrap), and only the df column is uniformly NaN there. 'event_study' is a PRUNED-IF-PAYLOAD kit recompute (see the AggregationKit docstring variant): per-(g,t) O(n_treated+n_control) IF-ingredient arrays + unit-level arrays + the PANEL-LEVEL resolved survey design (on replicate designs the (n_obs x R) replicate matrix rides along - a unit-level collapse was reviewed and declined for verbatim-move safety); no panel data columns and no raw unit identifiers are retained; bootstrap fits get a SCALARS-ONLY kit and the ES route fails closed (NotImplementedError naming the fit-time route / n_bootstrap=0 refit; ContinuousDiDResults has no bootstrap_results field, so the config gate is the honest one). Replicate-weight designs ARE supported post-fit (IF-based compute_replicate_if_variance - no refit replay, the contrast with M-021/M-022). Fit-faithful quirk: empty-post_gt fits leave ES rows at NaN inference on both routes (has_post_cells flag). CONTAINER SHAPES: simple = 2 rows (targets att/acrt - the dual-estimand case the AggregationResult target column exists for; n = disjoint treated+control units total, n_kind='units'); dose = 2N target-discriminated rows (labels = the dose grid twice; n NaN / n_kind None / weight None - grid evaluation points carry no count or mass); df from the stored dose_response_att.df_survey channel (finite-and->0 else NaN; the raw stored value incl. the replicate 0-sentinel feeds the to_dataframe-exact t/p derivation). Ships the FIRST heterogeneous-target container, with the AggregationResult summary()/to_dataframe rendering amendment (target column + neutral estimate heading when targets mixed; FIRST-APPEARANCE target-block ordering, labels ascending within block under the _sortable guard; uniform-target producers byte-stable) - normative rule in v4-design section 6. Supported subset simple|event_study|dose - group/calendar fail closed via the mixin; balance_e applies to no level (empty _AGGREGATE_BALANCE_E_TYPES); weights= rejected. Admission: honest/pretrends containers rejected BY DESIGN (no joint ES covariance; bins not reference-normalized - see M-093). Bootstrap-ES post-fit replay is the TODO.md row. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons)." + notes: "REMOVED at the 4.0 cut (PR 3c, 2026-10-21). Shimmed in 3.9: fit(aggregate=) warns via the shared NOT_SUPPLIED sentinel (a plain fit() never warns; supplying ANY value incl. None warns once, then the legacy routing runs unchanged). Unlike the EfficientDiD/Imputation shims, fit-time VALUE VALIDATION pre-existed and STAYS: unknown strings still raise ValueError after the warning ((None, 'dose', 'eventstudy') only). The no-underscore 'eventstudy' spelling dies with the param in 4.0; aggregate() accepts only the unified vocabulary + 'dose' as this estimator's documented extra level. MIXED VIEW/RECOMPUTE architecture (unique in the register): 'simple' and 'dose' are pure VIEWS over stored public fields - the dose curves and the overall binarized ATT (ATT^{loc} under PT; equals ATT^{glob} under SPT) plus ACRT^{glob} are ALWAYS computed by fit (aggregate='dose' was a fit-time no-op) - so both levels are PERMITTED on bootstrap fits (the library-wide per-level relay rule, since [M-027] converged CS/EDiD/Imputation/TwoStage onto it), relaying stored inference verbatim: the overall rows carry the FINITE safe_inference t fit stores beside percentile p/CI, the dose rows reproduce DoseResponseCurve.to_dataframe (NaN t under bootstrap), and only the df column is uniformly NaN there. 'event_study' is a PRUNED-IF-PAYLOAD kit recompute (see the AggregationKit docstring variant): per-(g,t) O(n_treated+n_control) IF-ingredient arrays + unit-level arrays + the PANEL-LEVEL resolved survey design (on replicate designs the (n_obs x R) replicate matrix rides along - a unit-level collapse was reviewed and declined for verbatim-move safety); no panel data columns and no raw unit identifiers are retained; bootstrap fits get a SCALARS-ONLY kit and the ES route fails closed (NotImplementedError naming the fit-time route / n_bootstrap=0 refit; ContinuousDiDResults has no bootstrap_results field, so the config gate is the honest one). Replicate-weight designs ARE supported post-fit (IF-based compute_replicate_if_variance - no refit replay, the contrast with M-021/M-022). Fit-faithful quirk: empty-post_gt fits leave ES rows at NaN inference on both routes (has_post_cells flag). CONTAINER SHAPES: simple = 2 rows (targets att/acrt - the dual-estimand case the AggregationResult target column exists for; n = disjoint treated+control units total, n_kind='units'); dose = 2N target-discriminated rows (labels = the dose grid twice; n NaN / n_kind None / weight None - grid evaluation points carry no count or mass); df from the stored dose_response_att.df_survey channel (finite-and->0 else NaN; the raw stored value incl. the replicate 0-sentinel feeds the to_dataframe-exact t/p derivation). Ships the FIRST heterogeneous-target container, with the AggregationResult summary()/to_dataframe rendering amendment (target column + neutral estimate heading when targets mixed; FIRST-APPEARANCE target-block ordering, labels ascending within block under the _sortable guard; uniform-target producers byte-stable) - normative rule in v4-design section 6. Supported subset simple|event_study|dose - group/calendar fail closed via the mixin; balance_e applies to no level (empty _AGGREGATE_BALANCE_E_TYPES); weights= rejected. Admission: honest/pretrends containers rejected BY DESIGN (no joint ES covariance; bins not reference-normalized - see M-093). Bootstrap-ES post-fit replay is the TODO.md row. DiagnosticReport now derives the event-study surface via post-fit aggregate('event_study') when the raw field is absent, so its ES-gated checks run on plain fits (derivation failures surface as explicit skip reasons). AT 4.0 (PR 3c): the 'eventstudy' spelling is gone with the param; fits with n_bootstrap > 0 ALWAYS compute the binarized event study at fit (the only route - the kit stays scalars-only there) and aggregate('event_study') RELAYS the stored surface (kit-snapshot alpha/anticipation/base_period/survey metadata; NaN df column) instead of raising; the analytic fit-time ES inference arm was deleted (dead: the surface was None whenever it ran). STORAGE RULE: the surface is stored IFF _run_bootstrap ran (post cells exist); a bootstrapped fit with no post-treatment cells clears it and the relay raises the shared absent-surface ValueError (previously NotImplementedError from the bootstrap gate). Materialization is FAIL-SOFT with a one-shot overall-only retry (warns once 'could not be materialized'; the overall/dose bootstrap quintets are unchanged on seeded fits; bootstrap failures still propagate as before). The ES aggregator's warnings now fire on bootstrapped plain fits; summary() prints and to_dataframe(level='event_study') returns the table there. Positional slot shifts: survey_design 8 -> 7, covariates 9 -> 8." - id: M-026 kind: param group: aggregate-postfit @@ -1026,7 +1026,7 @@ rows: status: planned phase: 5 code_refs: [diff_diff/staggered_results.py, diff_diff/sun_abraham.py, diff_diff/imputation_results.py, diff_diff/two_stage_results.py, diff_diff/stacked_did_results.py, diff_diff/efficient_did_results.py, diff_diff/continuous_did_results.py, diff_diff/wooldridge_results.py, diff_diff/chaisemartin_dhaultfoeuille_results.py, diff_diff/lpdid_results.py, diff_diff/staggered_triple_diff_results.py, diff_diff/results.py, diff_diff/had.py, diff_diff/visualization/_event_study.py, diff_diff/honest_did.py, diff_diff/pretrends.py] - notes: "4.0 sentinel retirement + schema enforcement (spec section 5): the n_groups==0 / n_obs==0 reference-row sentinels retire; every estimator's to_dataframe(level='event_study') emits the [M-092] column schema; the plotter / HonestDiD / PreTrendsPower consume the unified surface. Thirteen ES-carrying source modules enumerated file-by-file (results.py covers MultiPeriodDiD + SpilloverDiD) plus the plotter. behavior-at-done requires test_ref. PARTIAL PRE-CUT DELIVERY (2(b) PR-1, with M-026): the consumer half shipped for CS-SOURCED containers - compute_honest_did, compute_pretrends_power (both with SOURCE-SCOPED admission: source == CallawaySantAnnaResults only, rejecting dCDH l1 containers BY DESIGN since their placebo semantics need honest_did's native branch, and every other producer pending its own aggregate() migration) and plot_event_study / plot_honest_event_study (no source scoping - plotting is label-faithful). Admission widening is each later shim PR's methodology decision, not automatic. SECOND PRE-CUT AMENDMENT (2(b) PR-2, with M-024): admission widened to STACKED-SOURCED containers in compute_honest_did + compute_pretrends_power (source in {CallawaySantAnnaResults, StackedDiDResults}; kappa_pre >= 2 required for estimated pre-periods; honest additionally needs a non-singular full retained event-study covariance (pre+post sub-block); withheld-inference rows admitted with a source-scoped warning; the seven producer-derived guard messages now derive the producer from surface.source). Source scoping continues for all other producers. THIRD PRE-CUT AMENDMENT (2(b) PR-3a, with M-023): admission NOT widened to EfficientDiD-sourced containers BY DESIGN (the dCDH precedent) - the PRIMARY ground is the absent joint event-study covariance (container vcov=None; per-row df was all-NaN at that amendment and is finite on analytical survey fits since the 3.12 M-092 completion - the rejection ground is the missing joint covariance, unchanged); reference semantics are regime-dependent (PT-All: no reference row, universal first-period baseline; PT-Post: a materialized mechanical zero anchor at e=-1-anticipation when estimated, marked via the membership-gated reference_period property - never fabricated when absent); both terminal TypeErrors now state the EfficientDiD rejection explicitly; source scoping continues to reject. FOURTH PRE-CUT AMENDMENT (2(b) PR-3b, with M-021/M-022): admission NOT widened by the Imputation/TwoStage migrations - ImputationDiD rejected BY DESIGN (no joint ES covariance; per-horizon Theorem-3 conservative SEs only); TwoStageDiD DEFERRED despite carrying the joint Gardner-GMM covariance on analytical fits (M-092), because its pre-period coefficients are stage-1 residual means (the reference horizon is dropped from the no-intercept Stage-2 design; the zero anchor row is appended mechanically), not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the delta_0=0 normalization into its boundary/bridge constraints - admission awaits a normalization derivation (DEFERRED.md paper-gated row); both terminal TypeErrors state the per-producer grounds. FIFTH PRE-CUT AMENDMENT (2(b) PR-3c, with M-025): admission NOT widened by the ContinuousDiD migration - rejected BY DESIGN on two independent grounds: no joint event-study covariance exists (per-bin IF SEs only), and the binarized bins carry NO reference-period normalization at all (no reference row exists in the surface; each bin is a raw reweighted binarized ATT level), so HonestDiD's delta_0=0 arithmetic has no anchor even in principle without new methodology; both terminal TypeErrors name the ContinuousDiD rejection alongside EfficientDiD/Imputation. Amended pre-cut (2(b) PR-4, with M-027): HeterogeneousAdoptionDiD admission NOT widened by the HAD migration - DEFERRED, not by-design (the TwoStage class): its event-study coefficients ARE reference-normalized (each horizon differences against the F-1 anchor) but the anchor row is omitted from the container (identically zero AND the WAS is not identified there - no reference row exists for the consumer grid) and no joint cross-horizon covariance exists (per-horizon independent sandwiches; the DEFERRED.md row); both terminal TypeErrors name the HAD deferral in that corrected wording. Source scoping continues to reject all other producers. SIXTH PRE-CUT AMENDMENT (Phase 3(a), with M-010): admission widened to TWFE-SOURCED CALENDAR containers in compute_honest_did + compute_pretrends_power via a dedicated calendar route that reconstructs the native MultiPeriodDiDResults-branch inputs from the container's post_periods partition provenance (single-reference gate; HonestDiD additionally GEOMETRY-SCOPED to chronological partitions - suffix post_periods + last-pre reference - with the pre-existing native non-suffix geometry limitation documented in the REGISTRY HonestDiD Note and the DEFERRED.md transform-or-reject row; PreTrendsPower plumbs reference_period through the gamma-unit helper, string-label degradation included); relative-scale admission unchanged. SEVENTH PRE-CUT AMENDMENT (DML PR-B1): admission widened to DMLDiD-SOURCED containers in compute_honest_did + compute_pretrends_power (source in {CallawaySantAnnaResults, StackedDiDResults, DMLDiDResults}): DMLDiDResults subclasses CallawaySantAnnaResults and its aggregate('event_study') container is built by the INHERITED CS kit channel, so it carries the same staggered ATT(g,t) payload, reference semantics, base_period provenance, and (bootstrapped) replay behavior as a CS container - the same-semantics ground the original CS admission rests on; the producer-derived guard messages treat DMLDiDResults like CS (both expose base_period). DMLDiD adds NO builder (M-092 untouched) and never populates fit-time event_study_effects (its sentinel-retirement obligation is discharged by the shared CS code). Source scoping continues to reject all other producers. EIGHTH PRE-CUT AMENDMENT (4.0 cut PR 3b, with M-020/M-023/M-117/M-120): the fit-time aggregate=/balance_e= keywords are gone on CallawaySantAnna and EfficientDiD, so those two producers populate the legacy event_study_effects/group_effects fields on NO fit (analytic and bootstrapped; the post-fit replay is the only bootstrap route) and their to_dataframe(level='event_study'|'group') raises on every fresh fit until this row's schema rewire lands; admission is unchanged by the removals (the ImputationDiD/TwoStageDiD/ContinuousDiD removals follow in PR 3c). This row stays planned: its transition remains the 4.0 sentinel retirement + full-producer schema enforcement." + notes: "4.0 sentinel retirement + schema enforcement (spec section 5): the n_groups==0 / n_obs==0 reference-row sentinels retire; every estimator's to_dataframe(level='event_study') emits the [M-092] column schema; the plotter / HonestDiD / PreTrendsPower consume the unified surface. Thirteen ES-carrying source modules enumerated file-by-file (results.py covers MultiPeriodDiD + SpilloverDiD) plus the plotter. behavior-at-done requires test_ref. PARTIAL PRE-CUT DELIVERY (2(b) PR-1, with M-026): the consumer half shipped for CS-SOURCED containers - compute_honest_did, compute_pretrends_power (both with SOURCE-SCOPED admission: source == CallawaySantAnnaResults only, rejecting dCDH l1 containers BY DESIGN since their placebo semantics need honest_did's native branch, and every other producer pending its own aggregate() migration) and plot_event_study / plot_honest_event_study (no source scoping - plotting is label-faithful). Admission widening is each later shim PR's methodology decision, not automatic. SECOND PRE-CUT AMENDMENT (2(b) PR-2, with M-024): admission widened to STACKED-SOURCED containers in compute_honest_did + compute_pretrends_power (source in {CallawaySantAnnaResults, StackedDiDResults}; kappa_pre >= 2 required for estimated pre-periods; honest additionally needs a non-singular full retained event-study covariance (pre+post sub-block); withheld-inference rows admitted with a source-scoped warning; the seven producer-derived guard messages now derive the producer from surface.source). Source scoping continues for all other producers. THIRD PRE-CUT AMENDMENT (2(b) PR-3a, with M-023): admission NOT widened to EfficientDiD-sourced containers BY DESIGN (the dCDH precedent) - the PRIMARY ground is the absent joint event-study covariance (container vcov=None; per-row df was all-NaN at that amendment and is finite on analytical survey fits since the 3.12 M-092 completion - the rejection ground is the missing joint covariance, unchanged); reference semantics are regime-dependent (PT-All: no reference row, universal first-period baseline; PT-Post: a materialized mechanical zero anchor at e=-1-anticipation when estimated, marked via the membership-gated reference_period property - never fabricated when absent); both terminal TypeErrors now state the EfficientDiD rejection explicitly; source scoping continues to reject. FOURTH PRE-CUT AMENDMENT (2(b) PR-3b, with M-021/M-022): admission NOT widened by the Imputation/TwoStage migrations - ImputationDiD rejected BY DESIGN (no joint ES covariance; per-horizon Theorem-3 conservative SEs only); TwoStageDiD DEFERRED despite carrying the joint Gardner-GMM covariance on analytical fits (M-092), because its pre-period coefficients are stage-1 residual means (the reference horizon is dropped from the no-intercept Stage-2 design; the zero anchor row is appended mechanically), not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the delta_0=0 normalization into its boundary/bridge constraints - admission awaits a normalization derivation (DEFERRED.md paper-gated row); both terminal TypeErrors state the per-producer grounds. FIFTH PRE-CUT AMENDMENT (2(b) PR-3c, with M-025): admission NOT widened by the ContinuousDiD migration - rejected BY DESIGN on two independent grounds: no joint event-study covariance exists (per-bin IF SEs only), and the binarized bins carry NO reference-period normalization at all (no reference row exists in the surface; each bin is a raw reweighted binarized ATT level), so HonestDiD's delta_0=0 arithmetic has no anchor even in principle without new methodology; both terminal TypeErrors name the ContinuousDiD rejection alongside EfficientDiD/Imputation. Amended pre-cut (2(b) PR-4, with M-027): HeterogeneousAdoptionDiD admission NOT widened by the HAD migration - DEFERRED, not by-design (the TwoStage class): its event-study coefficients ARE reference-normalized (each horizon differences against the F-1 anchor) but the anchor row is omitted from the container (identically zero AND the WAS is not identified there - no reference row exists for the consumer grid) and no joint cross-horizon covariance exists (per-horizon independent sandwiches; the DEFERRED.md row); both terminal TypeErrors name the HAD deferral in that corrected wording. Source scoping continues to reject all other producers. SIXTH PRE-CUT AMENDMENT (Phase 3(a), with M-010): admission widened to TWFE-SOURCED CALENDAR containers in compute_honest_did + compute_pretrends_power via a dedicated calendar route that reconstructs the native MultiPeriodDiDResults-branch inputs from the container's post_periods partition provenance (single-reference gate; HonestDiD additionally GEOMETRY-SCOPED to chronological partitions - suffix post_periods + last-pre reference - with the pre-existing native non-suffix geometry limitation documented in the REGISTRY HonestDiD Note and the DEFERRED.md transform-or-reject row; PreTrendsPower plumbs reference_period through the gamma-unit helper, string-label degradation included); relative-scale admission unchanged. SEVENTH PRE-CUT AMENDMENT (DML PR-B1): admission widened to DMLDiD-SOURCED containers in compute_honest_did + compute_pretrends_power (source in {CallawaySantAnnaResults, StackedDiDResults, DMLDiDResults}): DMLDiDResults subclasses CallawaySantAnnaResults and its aggregate('event_study') container is built by the INHERITED CS kit channel, so it carries the same staggered ATT(g,t) payload, reference semantics, base_period provenance, and (bootstrapped) replay behavior as a CS container - the same-semantics ground the original CS admission rests on; the producer-derived guard messages treat DMLDiDResults like CS (both expose base_period). DMLDiD adds NO builder (M-092 untouched) and never populates fit-time event_study_effects (its sentinel-retirement obligation is discharged by the shared CS code). Source scoping continues to reject all other producers. EIGHTH PRE-CUT AMENDMENT (4.0 cut PRs 3b/3c, with M-020..M-027/M-117..M-120/M-139): the fit-time aggregate=/balance_e= keywords are gone, which fixes WHICH producers populate the legacy event_study_effects/group_effects fields - CallawaySantAnna and EfficientDiD: None on EVERY fit (analytic and bootstrapped; the post-fit replay is the only bootstrap route); ImputationDiD and TwoStageDiD: None on analytic fits, ES + group dicts (percentile se/p/CI, df cleared) on bootstrapped fits IFF bootstrap_results is not None, relayed by aggregate(); ContinuousDiD: None on analytic fits, the ES dict on bootstrapped fits IFF _run_bootstrap ran, relayed by aggregate(); StackedDiD and SunAbraham: always populated (unchanged). CS/EDiD to_dataframe(level='event_study'|'group') therefore raises on every fresh fit until this row's schema rewire lands; admission is unchanged by the removals. This row stays planned: its transition remains the 4.0 sentinel retirement + full-producer schema enforcement." # ---- Behavior policies (schema-tracked, spec-governed; no reality probe) - - id: M-080 @@ -1433,12 +1433,12 @@ rows: introduced_in: "3.9" deprecated_in: "3.9" removed_in: "4.0" - status: shimmed + status: removed phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py code_refs: [diff_diff/imputation.py, diff_diff/imputation_aggregation.py, diff_diff/imputation_results.py, diff_diff/aggregation.py] - notes: "balance_e moves from fit() onto aggregate() with [M-021] (joint FutureWarning; the M-117/M-120 twin; the imputation_did wrapper forwards the sentinel). Applies to event-study aggregation only (the mixin default vocabulary), so aggregate(type='simple'|'group'|'total', balance_e=...) raises. ImputationDiD's balance rule is the BALANCED-WINDOW rule - a cohort is retained iff its observed relative-time set (checked against the FULL panel via _build_cohort_rel_times) covers the contiguous window [-balance_e, max_h] (imputation_aggregation.py _compute_balanced_cohort_mask) - divergent from CS/EfficientDiD's anchor-horizon rule ([M-117]/[M-120] document theirs; TwoStageDiD shares the balanced-window rule, [M-119]). A window no cohort satisfies warns (UserWarning) and yields the reference-marker-only dict -> a legal near-empty container." + notes: "REMOVED at the 4.0 cut (PR 3c, 2026-10-21). balance_e moves from fit() onto aggregate() with [M-021] (joint FutureWarning; the M-117/M-120 twin; the imputation_did wrapper forwards the sentinel). Applies to event-study aggregation only (the mixin default vocabulary), so aggregate(type='simple'|'group'|'total', balance_e=...) raises. ImputationDiD's balance rule is the BALANCED-WINDOW rule - a cohort is retained iff its observed relative-time set (checked against the FULL panel via _build_cohort_rel_times) covers the contiguous window [-balance_e, max_h] (imputation_aggregation.py _compute_balanced_cohort_mask) - divergent from CS/EfficientDiD's anchor-horizon rule ([M-117]/[M-120] document theirs; TwoStageDiD shares the balanced-window rule, [M-119]). A window no cohort satisfies warns (UserWarning) and yields the reference-marker-only dict -> a legal near-empty container. AT 4.0 (PR 3c): aggregate('event_study', balance_e=k) on a BOOTSTRAPPED ImputationDiD fit raises NotImplementedError (the relayed fit-time surface was computed with balance_e=None and the per-target draws are not retained; re-fit with n_bootstrap=0); analytic fits recompute with balance_e as before." - id: M-119 kind: param group: aggregate-postfit @@ -1447,12 +1447,12 @@ rows: introduced_in: "3.9" deprecated_in: "3.9" removed_in: "4.0" - status: shimmed + status: removed phase: 5 warning: FutureWarning test_ref: tests/test_aggregate_contract.py code_refs: [diff_diff/two_stage.py, diff_diff/two_stage_aggregation.py, diff_diff/two_stage_results.py, diff_diff/aggregation.py] - notes: "balance_e moves from fit() onto aggregate() with [M-022] (joint FutureWarning; the M-117/M-120 twin; the two_stage_did wrapper forwards the sentinel). Applies to event-study aggregation only (the mixin default vocabulary), so aggregate(type='simple'|'group'|'total', balance_e=...) raises. TwoStageDiD's balance rule is the BALANCED-WINDOW rule - a cohort is retained iff its observed relative-time set (checked against the full panel via _build_cohort_rel_times) covers the contiguous window [-balance_e, max_h] (two_stage_aggregation.py _stage2_event_study) - the ImputationDiD rule ([M-118]), divergent from CS/EfficientDiD's anchor-horizon rule. Zero qualifying cohorts warns ('No cohorts satisfy balance_e=...') and yields the reference-row-only dict with vcov=None -> a legal near-empty container." + notes: "REMOVED at the 4.0 cut (PR 3c, 2026-10-21). balance_e moves from fit() onto aggregate() with [M-022] (joint FutureWarning; the M-117/M-120 twin; the two_stage_did wrapper forwards the sentinel). Applies to event-study aggregation only (the mixin default vocabulary), so aggregate(type='simple'|'group'|'total', balance_e=...) raises. TwoStageDiD's balance rule is the BALANCED-WINDOW rule - a cohort is retained iff its observed relative-time set (checked against the full panel via _build_cohort_rel_times) covers the contiguous window [-balance_e, max_h] (two_stage_aggregation.py _stage2_event_study) - the ImputationDiD rule ([M-118]), divergent from CS/EfficientDiD's anchor-horizon rule. Zero qualifying cohorts warns ('No cohorts satisfy balance_e=...') and yields the reference-row-only dict with vcov=None -> a legal near-empty container. AT 4.0 (PR 3c): aggregate('event_study', balance_e=k) on a BOOTSTRAPPED TwoStageDiD fit raises NotImplementedError (the relayed fit-time surface was computed with balance_e=None and the per-level GMM scores are not retained; re-fit with n_bootstrap=0); analytic fits recompute with balance_e as before." - id: M-126 kind: behavior group: variance-cr1-convergence @@ -1530,7 +1530,7 @@ rows: phase: 2 test_ref: tests/test_variance_conventions.py code_refs: [diff_diff/utils.py, diff_diff/linalg.py, diff_diff/estimators.py, diff_diff/twfe.py, diff_diff/results.py, diff_diff/sun_abraham.py, diff_diff/wooldridge.py, diff_diff/wooldridge_results.py, diff_diff/stacked_did.py, diff_diff/stacked_did_results.py, diff_diff/imputation.py, diff_diff/imputation_results.py, diff_diff/lpdid.py, diff_diff/lpdid_results.py, docs/methodology/REGISTRY.md, docs/methodology/variance-conventions.md] - notes: "Tail-df consolidation (D4 + the normal-theory tail-df defect family of the 3.9 variance-consolidation program, PR C). df_convention becomes a THREE-VALUE knob {'residual','cluster','normal'} on DiD/MPD/TWFE/LinearRegression (previously two-value; existing values unchanged, M-004..M-006 old_default untouched; 'normal' is deliberate z at the fallback level on EVERY fit, unclustered and conley included) and is EXTENDED to SunAbraham / WooldridgeDiD(OLS arms) / StackedDiD / ImputationDiD(pretrends lead regression) / LPDiD via the new utils.resolve_tail_df fallback resolver (survey/replicate df and hc2_bm Bell-McCaffrey per-coefficient/contrast DOF keep precedence under every value; 'cluster' fails closed warn+NaN at G<=1; unclustered/conley fits are inert for 'cluster'; a non-positive residual df falls back to normal theory with a warning). DEFECT FIX shipped as the new 'residual' default: WooldridgeDiD default-hc1 (within AND the hc1+cohort_trends full-dummy arm), StackedDiD, SunAbraham aggregates (D4: one fit reported residual-t cells and normal-theory aggregates - aggregates now reuse the saturated regression's df_ so cells and aggregates share ONE df source), and ImputationDiD pretrends leads previously passed df=None (silent normal-theory z) on clustered analytical paths; at |t|=2 z understates the t(G-1) p-value by 24.2% at G=20. Under the residual default the numeric movement is z vs t(large residual df) - small but honest; the G-1 convergence with fixest/Stata is the 4.0 flip. 'normal' reproduces the pre-3.9 numbers on the fixed surfaces' affected lanes (on lanes already at t - unclustered DiD-family fits, Wooldridge classical/hc2, SA cells - it is a NEW deliberate z option, not a compatibility path). LPDiD keeps its Stata-anchored t(G-1) as default 'cluster' (bit-identical; already at the 4.0 target, no flip row; degenerate lanes - unclustered refit, RA G<=1, saturated early return - keep literal df=None under all values; RA-path 'residual' = n_total - k0_kept - 1, the pooled M-estimator parameter count, a library convention with no external anchor since Stata teffects ra reports z). Wooldridge logit/poisson arms (survey df or normal theory, knob-independent, explicit-non-default warns) and all IF-based estimators (CallawaySantAnna, TwoStageDiD, dCDH, EfficientDiD, ContinuousDiD, ChangesInChanges, SyntheticDiD, HAD - L3) plus OLS-based TripleDifference/TROP are out of the knob's scope. Wild-cluster-bootstrap p/CIs are test-inversion based and invariant. Provenance: df_convention echoed on the five standalone results containers; SunAbraham/StackedDiD gain inference_df (the overall-ATT df actually used; None under bootstrap); WooldridgeDiDResults._df_one_way renamed _df_analytic_fallback (now the resolved survey-first fallback on ALL OLS arms, live on bootstrap fits so post-fit aggregate() reproduces fit-time inference) with __setstate__ pickle migration. Defaults for the four new residual surfaces flip to 'cluster' at 4.0 (M-128..M-131, same policy as M-004..M-006). introduced_in gates the 3.9 cut; deprecated_in stays null so the early-flip guard does not fire against the PR that ships it. AMENDED (2(b) PR-3b, with M-021): the ImputationDiD inert-config warning predicate is REVISED from aggregate-keyed to REACHABILITY-based - warn iff df_convention is explicitly non-default AND the per-lead inference is unreachable for this fit config (reachable iff pretrends AND not replicate-weight AND (the deprecated fit-time aggregate='event_study'/'all' was supplied OR n_bootstrap <= 0)), because post-fit results.aggregate('event_study') made the old 'aggregate outside {event_study,all} is inert' claim false on analytical pretrends fits. Reachability-BASED, not exact: a fit whose bootstrap later FAILS (bootstrap_results=None) can still aggregate post-fit, so that corner warns spuriously - the warning fires before the bootstrap runs and cannot know." + notes: "Tail-df consolidation (D4 + the normal-theory tail-df defect family of the 3.9 variance-consolidation program, PR C). df_convention becomes a THREE-VALUE knob {'residual','cluster','normal'} on DiD/MPD/TWFE/LinearRegression (previously two-value; existing values unchanged, M-004..M-006 old_default untouched; 'normal' is deliberate z at the fallback level on EVERY fit, unclustered and conley included) and is EXTENDED to SunAbraham / WooldridgeDiD(OLS arms) / StackedDiD / ImputationDiD(pretrends lead regression) / LPDiD via the new utils.resolve_tail_df fallback resolver (survey/replicate df and hc2_bm Bell-McCaffrey per-coefficient/contrast DOF keep precedence under every value; 'cluster' fails closed warn+NaN at G<=1; unclustered/conley fits are inert for 'cluster'; a non-positive residual df falls back to normal theory with a warning). DEFECT FIX shipped as the new 'residual' default: WooldridgeDiD default-hc1 (within AND the hc1+cohort_trends full-dummy arm), StackedDiD, SunAbraham aggregates (D4: one fit reported residual-t cells and normal-theory aggregates - aggregates now reuse the saturated regression's df_ so cells and aggregates share ONE df source), and ImputationDiD pretrends leads previously passed df=None (silent normal-theory z) on clustered analytical paths; at |t|=2 z understates the t(G-1) p-value by 24.2% at G=20. Under the residual default the numeric movement is z vs t(large residual df) - small but honest; the G-1 convergence with fixest/Stata is the 4.0 flip. 'normal' reproduces the pre-3.9 numbers on the fixed surfaces' affected lanes (on lanes already at t - unclustered DiD-family fits, Wooldridge classical/hc2, SA cells - it is a NEW deliberate z option, not a compatibility path). LPDiD keeps its Stata-anchored t(G-1) as default 'cluster' (bit-identical; already at the 4.0 target, no flip row; degenerate lanes - unclustered refit, RA G<=1, saturated early return - keep literal df=None under all values; RA-path 'residual' = n_total - k0_kept - 1, the pooled M-estimator parameter count, a library convention with no external anchor since Stata teffects ra reports z). Wooldridge logit/poisson arms (survey df or normal theory, knob-independent, explicit-non-default warns) and all IF-based estimators (CallawaySantAnna, TwoStageDiD, dCDH, EfficientDiD, ContinuousDiD, ChangesInChanges, SyntheticDiD, HAD - L3) plus OLS-based TripleDifference/TROP are out of the knob's scope. Wild-cluster-bootstrap p/CIs are test-inversion based and invariant. Provenance: df_convention echoed on the five standalone results containers; SunAbraham/StackedDiD gain inference_df (the overall-ATT df actually used; None under bootstrap); WooldridgeDiDResults._df_one_way renamed _df_analytic_fallback (now the resolved survey-first fallback on ALL OLS arms, live on bootstrap fits so post-fit aggregate() reproduces fit-time inference) with __setstate__ pickle migration. Defaults for the four new residual surfaces flip to 'cluster' at 4.0 (M-128..M-131, same policy as M-004..M-006). introduced_in gates the 3.9 cut; deprecated_in stays null so the early-flip guard does not fire against the PR that ships it. AMENDED (2(b) PR-3b, with M-021): the ImputationDiD inert-config warning predicate is REVISED from aggregate-keyed to REACHABILITY-based - warn iff df_convention is explicitly non-default AND the per-lead inference is unreachable for this fit config (reachable iff pretrends AND not replicate-weight AND (the deprecated fit-time aggregate='event_study'/'all' was supplied OR n_bootstrap <= 0)), because post-fit results.aggregate('event_study') made the old 'aggregate outside {event_study,all} is inert' claim false on analytical pretrends fits. AMENDED AGAIN (4.0 cut PR 3c, with M-021): the fit-time keyword is gone and bootstrapped ImputationDiD fits now always materialize the pretrends lead inference at fit (relayed by aggregate()), so the predicate is simply pretrends AND not replicate-weight - exact on every fit config (the former bootstrap spurious-warning corner no longer exists)." - id: M-128 kind: default-flip group: df-convention-flip diff --git a/docs/v4-design.md b/docs/v4-design.md index 4469e603..72a4439f 100644 --- a/docs/v4-design.md +++ b/docs/v4-design.md @@ -631,11 +631,43 @@ provenance beside percentile statistics; a RECOMPUTE level on a bootstrapped fit either REPLAYS the fit-time bootstrap from retained RNG state (CS [M-020], EfficientDiD [M-023], DMLDiD via the inherited CS kit channel [M-093 seventh amendment]) or RAISES - it never silently -returns analytical inference. **Amendment (4.0 cut PR 3b, [M-020] [M-023] -[M-117] [M-120]):** on CallawaySantAnna and EfficientDiD the fit-time -`aggregate=`/`balance_e=` branches are deleted outright (the post-fit replay -exists) and their legacy `event_study_effects`/`group_effects` fields are None -after EVERY fit. **View-relay exception (Phase 2b PRs 1-2):** +returns analytical inference. **Amendment (4.0 cut PRs 3b/3c, the fit-time +`aggregate=`/`balance_e=` removals [M-020]..[M-027] [M-117]..[M-120] +[M-139]):** the deprecated keyword was the only route to bootstrap +event-study/group inference on ImputationDiD, TwoStageDiD and ContinuousDiD +(no per-draw state is retained for a replay), so its removal (PR 3c) RELAYS +that surface instead of closing it: a fit with `n_bootstrap > 0` on those +three estimators ALWAYS computes the fit-time bootstrap surfaces with the +pre-existing code (event study on all three, group on Imputation/TwoStage - +exactly what `aggregate="all"` computed, `balance_e=None`), stores them in +the legacy `event_study_effects`/`group_effects` fields, and +`aggregate('event_study'|'group')` RELAYS the stored dict with kit-snapshot +provenance (alpha, anticipation, survey metadata from the kit; the per-row df +column NaN, `vcov`/`vcov_index` None). `balance_e is not None` on such a fit +fails closed (`NotImplementedError`) - the stored surface was computed +unrestricted and cannot be re-masked. Analytic fits are unchanged (legacy +fields None, kit recompute on demand). Storage rule: the legacy fields hold a +surface IFF the fit carries bootstrap inference; on Imputation/TwoStage a +bootstrapped fit that ends analytical (zero valid draws, total bootstrap +failure) clears them and `aggregate()` recomputes analytically, while a fit +WITH bootstrap inference and no stored surface (pre-4.0 pickle, fail-soft +drop) raises the shared absent-surface error. ContinuousDiD's kit is +scalars-only on every bootstrapped fit, so there is no analytical fallback +there: a bootstrapped fit with no post-treatment cells never runs its +bootstrap (its overall ATT is already NaN with a `UserWarning`), stores no +surface, and `aggregate('event_study')` raises the absent-surface error +rather than recomputing. Materialization is FAIL-SOFT with a one-shot +overall-only retry: an exception while building the surfaces, or a bootstrap +attempt that raises/returns `None` with surfaces present, leaves them None, +warns once, and re-runs the bootstrap overall-only (a seeded fit reproduces +attempt 1's weight draws), so a plain bootstrapped fit's headline quintet and +its success/abort outcome are unchanged by the removal. On CallawaySantAnna +and EfficientDiD the fit-time branches are deleted outright (PR 3b; the +post-fit replay exists) and their legacy fields are None after EVERY fit. +Measured cost of the always-on surfaces (199 draws, 300 units x 12 periods): +Imputation 0.01s -> 0.08s, TwoStage 0.01s -> 0.04s, Continuous 0.05s -> +0.05s - a few hundredths of a second per small bootstrapped fit; all three +default to `n_bootstrap=0`. **View-relay exception (Phase 2b PRs 1-2):** estimators whose `aggregate()` RELAYS stored fields without recomputation need no influence-function kit - there is nothing to re-weight. The retention requirement binds RECOMPUTING estimators (the CallawaySantAnna @@ -644,7 +676,13 @@ verbatim, and StackedDiD [M-024] always materializes its event-study surface at fit (the pooled regression always includes the interactions) so both its levels are pure views - which is also why bootstrap-style fail-closing does not apply to them: every stored inference mode relays -faithfully. Estimators whose +faithfully. ImputationDiD [M-021], TwoStageDiD [M-022] and ContinuousDiD +[M-025] are HYBRIDS since the 4.0 cut (PR 3c): they RECOMPUTE from their kits +on analytic fits and RELAY the fit-materialized surface on bootstrapped fits, +so the retention requirement binds their analytic path only - the relayed +dict is read from the mutable public field (the StackedDiD view precedent), +with only alpha/anticipation/survey metadata taken from kit snapshots. +Estimators whose estimand is already a single aggregation (SunAbraham's saturated event study, LPDiD's per-horizon design) expose `aggregate()` where meaningful as additive surface; their native params (`only_event`/`only_pooled` etc.) are documented @@ -1031,7 +1069,7 @@ tests-first PR that flips no rows). already exists (CallawaySantAnna, EfficientDiD, StackedDiD, dCDH, HAD, the HAD pretest workflow) (SHIPPED, PR 3b). - 3c - the ImputationDiD/TwoStageDiD/ContinuousDiD removals plus the - bootstrapped-fit relay (section 6 amendment). + bootstrapped-fit relay (section 6 amendment) (SHIPPED, PR 3c). 4. The three merges: the MultiPeriodDiD class, `MultiPeriodDiDResults`, `PeriodEffect` and `EventStudy`; the QDiD class and `QDiDResults`; the `StaggeredTripleDifference` CLASS and `SDDD` - but NOT diff --git a/tests/_capture_v4_aggregate_oracles.py b/tests/_capture_v4_aggregate_oracles.py index b99137c9..4307732d 100644 --- a/tests/_capture_v4_aggregate_oracles.py +++ b/tests/_capture_v4_aggregate_oracles.py @@ -1,9 +1,9 @@ """Capture the pre-removal aggregation oracles for tests/test_aggregate_contract.py. -HISTORICAL since the 4.0 cut (PR 3b): this script drives the REMOVED fit-time -``aggregate=`` / ``balance_e=`` route on CallawaySantAnna / EfficientDiD / StackedDiD -/ dCDH / HAD (and imports the contract module's retired ``_boot_fit_time`` / -``_efficient_boot_fit_time`` helpers), so it is runnable only at the recorded +HISTORICAL since the 4.0 cut (PR 3b, completed by PR 3c): this script drives the +REMOVED fit-time ``aggregate=`` / ``balance_e=`` route on every adopter (and imports +the contract module's retired ``_boot_fit_time`` / ``_efficient_boot_fit_time`` +helpers), so it is runnable only at the recorded ``provenance.captured_at_head`` of ``tests/fixtures/v4_aggregate_oracles.json`` (``8724fecd``, whose ``diff_diff/`` is byte-identical to the capture tree) - the same convention as ``tests/_capture_v4_merge_ddd_oracles.py``. It is kept as the diff --git a/tests/helpers/legacy_surface.py b/tests/helpers/legacy_surface.py index 13439855..2d66bbeb 100644 --- a/tests/helpers/legacy_surface.py +++ b/tests/helpers/legacy_surface.py @@ -1,7 +1,6 @@ """Test helpers bridging the post-fit aggregation containers and the 3.x dict shapes. -The post-fit ``results.aggregate(...)`` container is the canonical route (the 3.9 -fit-time ``fit(aggregate=...)`` keyword is deprecated and removed at the 4.0 cut), so a +Since the 4.0 cut (PRs 3b/3c) the fit-time ``fit(aggregate=...)`` route is gone, so a test that used to read ``results.event_study_effects[h]["effect"]`` after a fit-time aggregation reads the post-fit container instead:: @@ -27,7 +26,7 @@ (``event_study_effects`` + ``event_study_vcov``/``_index`` + the surface's ``reference_event_times`` + ``cband_crit_value`` + ``event_study_df``), i.e. what a 3.x ``fit(aggregate="event_study", balance_e=...)`` stored. Use it ONLY - where a test exercises the native route that plain fits do not + where a test exercises the native route that live 4.0 fits no longer populate (plotter / HonestDiD / PreTrendsPower raw-field branches, ``build_event_study_surface(native)`` fixtures). diff --git a/tests/test_aggregate_contract.py b/tests/test_aggregate_contract.py index 43dd2079..fb3bd4f6 100644 --- a/tests/test_aggregate_contract.py +++ b/tests/test_aggregate_contract.py @@ -1,34 +1,33 @@ """Behavioral contract for post-fit ``results.aggregate()`` (spec section 6). The ``test_ref`` for every aggregate-postfit ledger row this file pins: -M-020/M-023/M-021/M-022 (the CS / EfficientDiD fit-time ``aggregate=`` removals -- 4.0 PR 3b - and the Imputation / TwoStage shims, deprecated since 3.9), -M-024/M-026 (the Stacked / dCDH removals + view relays), M-025 (the ContinuousDiD shim + MIXED -view/recompute aggregate() - 'simple'/'dose' views, 'event_study' -pruned-IF-payload kit on analytic fits), M-117/M-120/M-118/M-119 (``balance_e`` -lives on ``aggregate()`` only), M-027/M-139 (the HAD fit + pretest-workflow -mode-selector removals with panel-shape inference, and the two PURE-VIEW results -classes) and M-122 (``AggregationResult``). +M-020/M-023/M-021/M-022 (the CS / EfficientDiD / Imputation / TwoStage +fit-time ``aggregate=`` removals - 4.0 PRs 3b/3c), M-024/M-026 (the Stacked / dCDH +removals + view relays), M-025 (the ContinuousDiD removal + MIXED +view/relay/recompute aggregate() - 'simple'/'dose' views, 'event_study' +pruned-IF-payload kit on analytic fits, RELAY of the fit-time percentile +surface on bootstrapped fits), M-117/M-120/M-118/M-119 (``balance_e`` lives on +``aggregate()`` only), M-027/M-139 (the HAD fit + pretest-workflow +mode-selector removals with panel-shape inference, and the two PURE-VIEW +results classes) and M-122 (``AggregationResult``). The headline gate is ORACLE PARITY: the deprecated fit-time route that used to -serve as this file's numerical reference is gone on CallawaySantAnna / EfficientDiD -/ StackedDiD / dCDH / HAD (PR 3b) and scheduled for removal on ImputationDiD / -TwoStageDiD / ContinuousDiD (PR 3c), so every aggregation surface is pinned against -``tests/fixtures/v4_aggregate_oracles.json`` - captured on the unmodified tree by -``tests/_capture_v4_aggregate_oracles.py`` while that route still existed (see -``tests/fixtures/README.md``). -Analytic surfaces must match at ``rtol=1e-12, atol=1e-13``; bootstrapped -surfaces (the CS/EfficientDiD replays and the plain-fit overall quintets) match -the backend-matched oracle at the replay-parity tolerances (``rtol=1e-13, -atol=1e-13`` on att/se/t/CI, ``atol=2/n_bootstrap`` on the count-statistic -p-values). Drift here means a regression, not a design change. - -Since PR 3b the fit-time keywords are gone outright on CallawaySantAnna, -EfficientDiD, StackedDiD, dCDH and HAD (and ``did_had_pretest_workflow``): -``fit(aggregate=...)`` / ``fit(balance_e=...)`` raise Python's native ``TypeError`` -(the ``Test*Removal`` classes). The remaining ``Test*Shim`` classes pin the -Imputation/TwoStage/ContinuousDiD keywords' 3.9 FutureWarning behaviour until PR 3c -removes them together with the bootstrapped-fit relay. +serve as this file's numerical reference is gone, so every aggregation surface +is pinned against ``tests/fixtures/v4_aggregate_oracles.json`` - captured on the +unmodified tree by ``tests/_capture_v4_aggregate_oracles.py`` while that route +still existed (see ``tests/fixtures/README.md``). Analytic surfaces must match +at ``rtol=1e-12, atol=1e-13``; bootstrapped surfaces (the CS/EfficientDiD +replays, the Imputation/TwoStage/ContinuousDiD relays and the plain-fit overall +quintets) match the backend-matched oracle at the replay-parity tolerances +(``rtol=1e-13, atol=1e-13`` on att/se/t/CI, ``atol=2/n_bootstrap`` on the +count-statistic p-values). Drift here means a regression, not a design change. + +Since 4.0 the fit-time keywords are gone outright: ``fit(aggregate=...)`` / +``fit(balance_e=...)`` raise Python's native ``TypeError`` (pinned per +estimator below), and bootstrapped Imputation/TwoStage/ContinuousDiD fits +materialize their event-study (+ group) surfaces at fit - FAIL-SOFT - so +``aggregate()`` can relay them (the fail-soft path is pinned by failure +injection). """ import copy @@ -78,6 +77,17 @@ def _unexpected(old: str) -> str: #: (a draw within a ULP of the point estimate can flip one count), hence the #: additional ``atol=2/n_bootstrap`` applied by the callers. BOOT_TOL = dict(rtol=1e-13, atol=1e-13) +#: t-statistics are the RATIO att/se of two columns each bounded by ``rtol``, +#: so their relative drift compounds to ``2 * rtol`` to first order (the +#: Rust backend's OpenBLAS / pure-Rust paths on Linux and Windows sit at +#: 1.6e-13 against the macOS capture on the TwoStage Prop-5 bootstrap arm +#: while att and se each pass at 1e-13). ``_t_rtol`` applies that bound. +T_STAT_RTOL_FACTOR = 2.0 + + +def _t_rtol(rtol): + """Tolerance for a t-statistic derived from ``rtol``-bounded att and se.""" + return T_STAT_RTOL_FACTOR * rtol def _oracle_key(label): @@ -151,7 +161,7 @@ def _assert_es_matches_oracle(es, orow, *, rtol, atol, p_atol=None, name=""): tag = f"{name}/es[{e}]" _close(es.att[i], r["effect"], rtol=rtol, atol=atol, what=f"{tag}.att") _close(es.se[i], r["se"], rtol=rtol, atol=atol, what=f"{tag}.se") - _close(es.t_stat[i], r["t_stat"], rtol=rtol, atol=atol, what=f"{tag}.t_stat") + _close(es.t_stat[i], r["t_stat"], rtol=_t_rtol(rtol), atol=atol, what=f"{tag}.t_stat") _close(es.p_value[i], r["p_value"], rtol=rtol, atol=p_atol, what=f"{tag}.p_value") _close(es.conf_int_lower[i], r["ci_lo"], rtol=ci_rtol, atol=atol, what=f"{tag}.ci_lo") _close(es.conf_int_upper[i], r["ci_hi"], rtol=ci_rtol, atol=atol, what=f"{tag}.ci_hi") @@ -208,7 +218,7 @@ def _assert_group_matches_oracle(agg, orow, *, rtol, atol, p_atol=None, name="") tag = f"{name}/group[{g}]" _close(agg.att[i], r["effect"], rtol=rtol, atol=atol, what=f"{tag}.att") _close(agg.se[i], r["se"], rtol=rtol, atol=atol, what=f"{tag}.se") - _close(agg.t_stat[i], r["t_stat"], rtol=rtol, atol=atol, what=f"{tag}.t_stat") + _close(agg.t_stat[i], r["t_stat"], rtol=_t_rtol(rtol), atol=atol, what=f"{tag}.t_stat") _close(agg.p_value[i], r["p_value"], rtol=rtol, atol=p_atol, what=f"{tag}.p_value") _close(agg.conf_int_lower[i], r["ci_lo"], rtol=ci_rtol, atol=atol, what=f"{tag}.ci_lo") _close(agg.conf_int_upper[i], r["ci_hi"], rtol=ci_rtol, atol=atol, what=f"{tag}.ci_hi") @@ -232,7 +242,7 @@ def _assert_overall_matches_oracle(res, o, *, rtol, atol, p_atol=None, name=""): p_atol = atol if p_atol is None else p_atol _close(res.overall_att, o["att"], rtol=rtol, atol=atol, what=f"{name}.overall_att") _close(res.overall_se, o["se"], rtol=rtol, atol=atol, what=f"{name}.overall_se") - _close(res.overall_t_stat, o["t_stat"], rtol=rtol, atol=atol, what=f"{name}.overall_t") + _close(res.overall_t_stat, o["t_stat"], rtol=_t_rtol(rtol), atol=atol, what=f"{name}.overall_t") _close(res.overall_p_value, o["p_value"], rtol=rtol, atol=p_atol, what=f"{name}.overall_p") ci = res.overall_conf_int ci_rtol = max(rtol, CI_RTOL) @@ -1341,7 +1351,13 @@ def _assert_level_matches_oracle(post, orow, level, name=""): o = orow["overall"] assert np.allclose(got.att[0], got.n[0] * oracle_value(o["att"]), rtol=1e-12) assert np.allclose(got.se[0], got.n[0] * oracle_value(o["se"]), rtol=1e-12) - _close(got.t_stat[0], o["t_stat"], **ANALYTIC_TOL, what=f"{name}/total.t") + _close( + got.t_stat[0], + o["t_stat"], + rtol=_t_rtol(ANALYTIC_TOL["rtol"]), + atol=ANALYTIC_TOL["atol"], + what=f"{name}/total.t", + ) return if level == "simple": _assert_simple_relays_stored_overall(post) @@ -3367,7 +3383,7 @@ def test_hausman_pretest_emits_no_future_warning(self, efficient_panel): # --------------------------------------------------------------------------- # -# ImputationDiD (rows M-021/M-118): fit(aggregate=/balance_e=) shim pins + the +# ImputationDiD (rows M-021/M-118): fit(aggregate=/balance_e=) removal pins + the # PANEL-BACKED recompute aggregate() (kit refs = the _fit_data objects) on # analytic fits; bootstrapped fits RELAY the fit-time percentile surfaces # --------------------------------------------------------------------------- # @@ -3450,12 +3466,6 @@ def imputation_fitted(imputation_panel): return _fit_imputation(imputation_panel) -@pytest.fixture(scope="module") -def imputation_fit_time(imputation_panel): - """Deprecated fit-time aggregate="all" - the analytical inertness reference.""" - return _fit_imputation(imputation_panel, aggregate="all") - - def _im_oracle(arm): return load_oracles()["analytic"]["ImputationDiD"][arm] @@ -3468,7 +3478,52 @@ def _im_boot_oracle(arm): _IM_P_ATOL = 2.0 / _IM_BOOT_KW["n_bootstrap"] -class TestImputationShim: +def _assert_relay_matches_stored_surfaces(res, es, grp, *, has_vcov_fields=True): + """The relay IS the stored fit-time surface: ``aggregate('event_study')`` + equals ``build_event_study_surface(res)`` frame-for-frame and the stored + legacy dict entries row-for-row; ``aggregate('group')`` equals the stored + ``group_effects``; percentile provenance (NaN df, no joint vcov).""" + _frame_equal(es.to_dataframe(), build_event_study_surface(res).to_dataframe()) + assert es.vcov is None and es.vcov_index is None + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) + stored = res.event_study_effects + ref = np.asarray(es.is_reference, dtype=bool) + for i, e in enumerate(es.event_time): + row = stored[int(e)] + if ref[i]: + assert row["effect"] == 0.0 and row.get("n_obs", 0) == 0 + continue + for col, key in ( + ("att", "effect"), + ("se", "se"), + ("t_stat", "t_stat"), + ("p_value", "p_value"), + ): + got, want = float(getattr(es, col)[i]), float(row[key]) + assert got == want or (np.isnan(got) and np.isnan(want)), (e, col) + assert (float(es.conf_int_lower[i]), float(es.conf_int_upper[i])) == tuple( + float(x) for x in row["conf_int"] + ) or np.isnan(row["conf_int"][0]) + assert np.all(np.isnan(np.asarray(grp.df, dtype=float))) + for i, g in enumerate(grp.label): + row = res.group_effects[g] + for col, key in ( + ("att", "effect"), + ("se", "se"), + ("t_stat", "t_stat"), + ("p_value", "p_value"), + ): + got, want = float(getattr(grp, col)[i]), float(row[key]) + assert got == want or (np.isnan(got) and np.isnan(want)), (g, col) + assert row.get("df_used") is None + if has_vcov_fields: + assert res.event_study_vcov is None and res.event_study_vcov_index is None + assert res.event_study_df is None + + +class TestImputationRemoval: + """M-021 / M-118 removed.""" + def test_plain_fit_does_not_warn(self, imputation_panel): from diff_diff import ImputationDiD @@ -3477,45 +3532,32 @@ def test_plain_fit_does_not_warn(self, imputation_panel): ImputationDiD().fit(imputation_panel, **IMPUTATION_KW) assert [w for w in caught if issubclass(w.category, FutureWarning)] == [] - def test_aggregate_kwarg_warns_even_at_none(self, imputation_panel): - from diff_diff import ImputationDiD - - with pytest.warns(FutureWarning, match=r"ImputationDiD\.fit\(aggregate=\)"): - ImputationDiD().fit(imputation_panel, **IMPUTATION_KW, aggregate=None) - - def test_balance_e_kwarg_warns_alone(self, imputation_panel): + @pytest.mark.parametrize("kw", ["aggregate", "balance_e"]) + def test_removed_keyword_is_a_type_error(self, imputation_panel, kw): from diff_diff import ImputationDiD - with pytest.warns(FutureWarning, match=r"ImputationDiD\.fit\(balance_e=\)"): - ImputationDiD().fit(imputation_panel, **IMPUTATION_KW, balance_e=None) + with pytest.raises(TypeError, match=_unexpected(kw)): + ImputationDiD().fit(imputation_panel, **IMPUTATION_KW, **{kw: None}) + value = "event_study" if kw == "aggregate" else 0 + with pytest.raises(TypeError, match=_unexpected(kw)): + ImputationDiD().fit(imputation_panel, **IMPUTATION_KW, **{kw: value}) - def test_joint_supply_warns_once_naming_both(self, imputation_panel): + def test_signature_and_seventh_positional_is_survey_design(self): from diff_diff import ImputationDiD - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - ImputationDiD().fit( - imputation_panel, **IMPUTATION_KW, aggregate="event_study", balance_e=0 + params = list(inspect.signature(ImputationDiD.fit).parameters) + assert "aggregate" not in params and "balance_e" not in params + assert params[7] == "survey_design" + d, _ = _imputation_survey_panel() + sd = _imputation_survey_design() + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + positional = ImputationDiD().fit( + d, "outcome", "unit", "period", "first_treat", None, sd ) - fw = [w for w in caught if issubclass(w.category, FutureWarning)] - assert len(fw) == 1 - msg = str(fw[0].message) - assert "aggregate=" in msg and "balance_e=" in msg - - def test_unknown_string_still_acts_like_none(self, imputation_panel): - res = _fit_imputation(imputation_panel, aggregate="nonsense") - plain = _fit_imputation(imputation_panel) - assert res.event_study_effects is None and res.group_effects is None - assert res.overall_att == plain.overall_att - - def test_warn_and_still_work(self, imputation_fitted, imputation_fit_time): - assert imputation_fit_time.event_study_effects is not None - assert imputation_fit_time.group_effects is not None - es = imputation_fitted.aggregate("event_study") - for e, row in imputation_fit_time.event_study_effects.items(): - i = list(es.event_time).index(e) - if np.isfinite(row["effect"]) and not es.is_reference[i]: - np.testing.assert_allclose(row["effect"], es.att[i], rtol=1e-14) + keyword = ImputationDiD().fit(d, survey_design=sd, **IMPUTATION_KW) + assert positional.survey_metadata is not None + assert positional.overall_se == keyword.overall_se class TestImputationAggregate: @@ -3722,16 +3764,213 @@ def test_repeated_and_order_independent(self, imputation_fitted): b = imputation_fitted.aggregate("group").to_dataframe() pd.testing.assert_frame_equal(a, b) - # ---------------- bootstrapped fits (fail closed until the PR 3c relay) ---------------- + # ---------------- bootstrapped fits: the RELAY (4.0 PR 3c) ---------------- - def test_bootstrap_recompute_levels_fail_closed(self, imputation_panel): - res = _fit_imputation(imputation_panel, est_kw={"n_bootstrap": 19, "seed": 1}) + def test_bootstrap_relays_fit_time_surfaces(self, imputation_panel): + res = _fit_imputation(imputation_panel, est_kw=_IM_BOOT_KW) + assert res.bootstrap_results is not None + es, grp = res.aggregate("event_study"), res.aggregate("group") + _assert_relay_matches_stored_surfaces(res, es, grp, has_vcov_fields=False) + orow = _im_boot_oracle("default_all") + _assert_es_matches_oracle(es, orow, **BOOT_TOL, p_atol=_IM_P_ATOL, name="imp/boot") + _assert_group_matches_oracle(grp, orow, **BOOT_TOL, p_atol=_IM_P_ATOL, name="imp/boot") + _assert_overall_matches_oracle( + res, _im_boot_oracle("default_plain_overall"), **BOOT_TOL, p_atol=_IM_P_ATOL + ) + _assert_overall_matches_oracle(res, orow["overall"], **BOOT_TOL, p_atol=_IM_P_ATOL) + _assert_bootstrap_p_values_in_unit_interval(es) + with pytest.raises(NotImplementedError, match="balance_e"): + res.aggregate("event_study", balance_e=1) + _assert_bootstrap_simple_relay(res) + + def test_bootstrap_pretrends_relay_matches_oracle(self, imputation_panel): + res = _fit_imputation(imputation_panel, est_kw={**_IM_BOOT_KW, "pretrends": True}) + es = res.aggregate("event_study") + _assert_es_matches_oracle( + es, _im_boot_oracle("pretrends_all"), **BOOT_TOL, p_atol=_IM_P_ATOL, name="imp/boot/pre" + ) + # Lead rows ride the lead regression (analytical) but the relay still + # publishes NO df (identical to the pre-removal bootstrapped surface). + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) + _assert_overall_matches_oracle( + res, _im_boot_oracle("pretrends_plain_overall"), **BOOT_TOL, p_atol=_IM_P_ATOL + ) + + def test_bootstrap_survey_tsl_relay_matches_oracle(self): + d, _ = _imputation_survey_panel() + res = _fit_imputation(d, est_kw=_IM_BOOT_KW, survey_design=_imputation_survey_design()) + orow = _im_boot_oracle("survey_tsl_all") + es = res.aggregate("event_study") + _assert_es_matches_oracle(es, orow, **BOOT_TOL, p_atol=_IM_P_ATOL, name="imp/boot/tsl") + _assert_group_matches_oracle( + res.aggregate("group"), orow, **BOOT_TOL, p_atol=_IM_P_ATOL, name="imp/boot/tsl" + ) + _assert_overall_matches_oracle( + res, _im_boot_oracle("survey_tsl_plain_overall"), **BOOT_TOL, p_atol=_IM_P_ATOL + ) + assert es.df_survey == res.survey_metadata.df_survey + + def test_legacy_fields_populated_iff_bootstrapped(self, imputation_panel, imputation_fitted): + assert imputation_fitted.event_study_effects is None + assert imputation_fitted.group_effects is None + boot = _fit_imputation(imputation_panel, est_kw=_IM_BOOT_KW) + assert boot.event_study_effects and boot.group_effects + assert boot.event_study_df is None + + def test_pre_4_0_pickle_without_surface_raises(self, imputation_panel): + """A 3.x bootstrapped plain fit stored no surface: the relay refuses + with the refit message instead of recomputing analytically.""" + res = _fit_imputation(imputation_panel, est_kw=_IM_BOOT_KW) + legacy = dataclasses.replace(res, event_study_effects=None, group_effects=None) + with pytest.raises(ValueError, match="carries no stored event_study surface"): + legacy.aggregate("event_study") + with pytest.raises(ValueError, match="carries no stored group surface"): + legacy.aggregate("group") + _assert_bootstrap_simple_relay(legacy) # the scalar relays still work + + def test_relay_is_a_view_of_the_public_field(self, imputation_panel): + """The relay payload reads the MUTABLE public field by definition + (StackedDiD precedent); only alpha/anticipation/survey metadata come + from kit snapshots.""" + res = _fit_imputation(imputation_panel, est_kw=_IM_BOOT_KW) + res.event_study_effects[0]["effect"] = 123.0 + es = res.aggregate("event_study") + assert float(es.att[list(es.event_time).index(0)]) == 123.0 + object.__setattr__(res, "alpha", 0.5) + assert res.aggregate("event_study").alpha == 0.05 + + # ---------------- fail-soft materialization (failure injection) ---------------- + + @staticmethod + def _boot_est(**extra): + from diff_diff import ImputationDiD + + return ImputationDiD(**_IM_BOOT_KW, **extra) + + def _assert_fail_soft_outcome(self, res, panel_kw): + """Surfaces dropped, bootstrap intact: the overall quintet is the + backend-matched PLAIN-fit oracle and the relay refuses.""" + assert res.event_study_effects is None and res.group_effects is None + assert res.bootstrap_results is not None + _assert_overall_matches_oracle( + res, _im_boot_oracle("default_plain_overall"), **BOOT_TOL, p_atol=_IM_P_ATOL + ) for level in ("event_study", "group"): - with pytest.raises(NotImplementedError, match="bootstrap") as exc: + with pytest.raises(ValueError, match="could not be materialized"): res.aggregate(level) - assert "aggregate('simple') and, where supported, aggregate('total') relay" in str( - exc.value + _assert_bootstrap_simple_relay(res) + + def test_failsoft_a_surface_hook_failure(self, imputation_panel, monkeypatch): + from diff_diff import ImputationDiD + + def boom(self, *a, **k): + raise RuntimeError("injected surface failure") + + monkeypatch.setattr(ImputationDiD, "_aggregate_event_study", boom) + with pytest.warns(UserWarning, match="could not be materialized") as rec: + res = self._boot_est().fit(imputation_panel, **IMPUTATION_KW) + assert any("injected surface failure" in str(w.message) for w in rec) + self._assert_fail_soft_outcome(res, IMPUTATION_KW) + + def test_failsoft_b_psi_stage_failure_retries_overall_only(self, imputation_panel, monkeypatch): + from diff_diff import ImputationDiD + + orig = ImputationDiD._precompute_bootstrap_psi + calls = [] + + def flaky(self, *a, **k): + calls.append(k.get("event_study_effects") is not None) + if k.get("event_study_effects") is not None: + raise RuntimeError("injected psi failure") + return orig(self, *a, **k) + + monkeypatch.setattr(ImputationDiD, "_precompute_bootstrap_psi", flaky) + with pytest.warns(UserWarning, match="could not be materialized"): + res = self._boot_est().fit(imputation_panel, **IMPUTATION_KW) + assert calls == [True, False] # attempt 1 with surfaces, ONE retry without + self._assert_fail_soft_outcome(res, IMPUTATION_KW) + + def test_failsoft_b2_total_psi_failure_falls_back_analytical( + self, imputation_panel, monkeypatch + ): + from diff_diff import ImputationDiD + + def boom(self, *a, **k): + raise RuntimeError("injected total psi failure") + + monkeypatch.setattr(ImputationDiD, "_precompute_bootstrap_psi", boom) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = self._boot_est().fit(imputation_panel, **IMPUTATION_KW) + msgs = [str(w.message) for w in caught] + assert any("Bootstrap pre-computation failed" in m for m in msgs) # today's fallback + assert not any("could not be materialized" in m for m in msgs) # no "unavailable" claim + assert res.bootstrap_results is None + assert res.event_study_effects is None and res.group_effects is None # storage rule + analytic = _fit_imputation(imputation_panel) + assert res.overall_se == analytic.overall_se # analytical overall + # ...and aggregate() RECOMPUTES analytically (no raise). + _frame_equal( + res.aggregate("event_study").to_dataframe(), + analytic.aggregate("event_study").to_dataframe(), + ) + _frame_equal( + res.aggregate("group").to_dataframe(), analytic.aggregate("group").to_dataframe() + ) + + def test_failsoft_b2_run_bootstrap_failure_on_retry_propagates( + self, imputation_panel, monkeypatch + ): + from diff_diff import ImputationDiD + + def boom(self, *a, **k): + raise RuntimeError("injected engine failure") + + monkeypatch.setattr(ImputationDiD, "_run_bootstrap", boom) + with pytest.raises(RuntimeError, match="injected engine failure"): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + self._boot_est().fit(imputation_panel, **IMPUTATION_KW) + + def test_failsoft_b3_n_valid_zero_bootstrapped_fit_recomputes(self): + """Storage-rule corner: every treated cell unidentified (no untreated + observations in any post period) -> n_valid == 0 skips the bootstrap + stage, the surfaces are None and aggregate() recomputes analytically.""" + rng = np.random.default_rng(0) + rows = [ + {"unit": u, "period": t, "first_treat": 4, "outcome": rng.normal() + 0.1 * t} + for u in range(30) + for t in range(1, 7) + ] + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = self._boot_est().fit(pd.DataFrame(rows), **IMPUTATION_KW) + assert np.isnan(res.overall_att) + assert res.bootstrap_results is None + assert res.event_study_effects is None and res.group_effects is None + assert not any("could not be materialized" in str(w.message) for w in caught) + es = res.aggregate("event_study") # recompute, no raise + assert len(es.event_time) > 0 + + def test_failsoft_c_real_trigger_lead_collinear_covariate(self, imputation_panel): + """The real trigger: pretrends=True + rank_deficient_action='error' + + a covariate equal to the lead-1 indicator makes the lead regression + raise ValueError (which the engine's (IndexError, LinAlgError) catch + does not cover). The plain fit SUCCEEDS with the fail-soft warning + (the pre-removal fit(aggregate='event_study') aborted).""" + from diff_diff import ImputationDiD + + d = imputation_panel.copy() + d["lead1"] = ((d["first_treat"] > 0) & (d["period"] == d["first_treat"] - 1)).astype(float) + with pytest.warns(UserWarning, match="could not be materialized"): + res = ImputationDiD(pretrends=True, rank_deficient_action="error", **_IM_BOOT_KW).fit( + d, covariates=["lead1"], **IMPUTATION_KW ) + assert res.bootstrap_results is not None + assert res.event_study_effects is None and res.group_effects is None + assert np.isfinite(res.overall_att) and np.isfinite(res.overall_se) + with pytest.raises(ValueError, match="could not be materialized"): + res.aggregate("event_study") def test_bootstrap_simple_relays_stored_quintet(self, imputation_panel): res = _fit_imputation(imputation_panel, est_kw={"n_bootstrap": 19, "seed": 1}) @@ -3804,14 +4043,15 @@ def test_group_df_used_relay(self): assert np.isnan(plain2.aggregate("group").to_dataframe()["df"].to_numpy()).all() def test_bootstrap_group_rows_clear_df_used(self, imputation_panel): - # Fit-time bootstrap override must never publish an analytical df - # beside percentile inference (public group_effects row dicts). - res = _fit_imputation( - imputation_panel, est_kw={"n_bootstrap": 19, "seed": 1}, aggregate="group" - ) + # The fit-time bootstrap override must never publish an analytical df + # beside percentile inference - on the STORED group_effects rows of a + # bootstrapped plain fit (populated since 4.0) and on the relay. + res = _fit_imputation(imputation_panel, est_kw=_IM_BOOT_KW) + assert res.group_effects for row in res.group_effects.values(): if np.isfinite(row["effect"]): assert row.get("df_used") is None + assert np.all(np.isnan(np.asarray(res.aggregate("group").df, dtype=float))) def test_empty_balance_window(self, imputation_fitted): with pytest.warns(UserWarning, match="no horizons"): @@ -3821,7 +4061,7 @@ def test_empty_balance_window(self, imputation_fitted): # --------------------------------------------------------------------------- # -# TwoStageDiD (rows M-022/M-119): fit(aggregate=/balance_e=) shim pins + the +# TwoStageDiD (rows M-022/M-119): fit(aggregate=/balance_e=) removal pins + the # PANEL-BACKED recompute aggregate() (column-subset working-frame kit) on # analytic fits; bootstrapped fits RELAY the fit-time percentile surfaces # --------------------------------------------------------------------------- # @@ -3917,11 +4157,6 @@ def twostage_fitted(twostage_panel): return _fit_twostage(twostage_panel) -@pytest.fixture(scope="module") -def twostage_fit_time(twostage_panel): - return _fit_twostage(twostage_panel, aggregate="all") - - def _ts_oracle(arm): return load_oracles()["analytic"]["TwoStageDiD"][arm] @@ -3934,7 +4169,9 @@ def _ts_boot_oracle(arm): _TS_P_ATOL = 2.0 / _TS_BOOT_KW["n_bootstrap"] -class TestTwoStageShim: +class TestTwoStageRemoval: + """M-022 / M-119 removed.""" + def test_plain_fit_does_not_warn(self, twostage_panel): from diff_diff import TwoStageDiD @@ -3943,43 +4180,30 @@ def test_plain_fit_does_not_warn(self, twostage_panel): TwoStageDiD().fit(twostage_panel, **TWOSTAGE_KW) assert [w for w in caught if issubclass(w.category, FutureWarning)] == [] - def test_aggregate_kwarg_warns_even_at_none(self, twostage_panel): - from diff_diff import TwoStageDiD - - with pytest.warns(FutureWarning, match=r"TwoStageDiD\.fit\(aggregate=\)"): - TwoStageDiD().fit(twostage_panel, **TWOSTAGE_KW, aggregate=None) - - def test_balance_e_kwarg_warns_alone(self, twostage_panel): + @pytest.mark.parametrize("kw", ["aggregate", "balance_e"]) + def test_removed_keyword_is_a_type_error(self, twostage_panel, kw): from diff_diff import TwoStageDiD - with pytest.warns(FutureWarning, match=r"TwoStageDiD\.fit\(balance_e=\)"): - TwoStageDiD().fit(twostage_panel, **TWOSTAGE_KW, balance_e=None) + with pytest.raises(TypeError, match=_unexpected(kw)): + TwoStageDiD().fit(twostage_panel, **TWOSTAGE_KW, **{kw: None}) + value = "event_study" if kw == "aggregate" else 0 + with pytest.raises(TypeError, match=_unexpected(kw)): + TwoStageDiD().fit(twostage_panel, **TWOSTAGE_KW, **{kw: value}) - def test_joint_supply_warns_once_naming_both(self, twostage_panel): + def test_signature_and_seventh_positional_is_survey_design(self): from diff_diff import TwoStageDiD - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - TwoStageDiD().fit(twostage_panel, **TWOSTAGE_KW, aggregate="event_study", balance_e=0) - fw = [w for w in caught if issubclass(w.category, FutureWarning)] - assert len(fw) == 1 - msg = str(fw[0].message) - assert "aggregate=" in msg and "balance_e=" in msg - - def test_unknown_string_still_acts_like_none(self, twostage_panel): - res = _fit_twostage(twostage_panel, aggregate="nonsense") - plain = _fit_twostage(twostage_panel) - assert res.event_study_effects is None and res.group_effects is None - assert res.overall_att == plain.overall_att - - def test_warn_and_still_work(self, twostage_fitted, twostage_fit_time): - assert twostage_fit_time.event_study_effects is not None - assert twostage_fit_time.group_effects is not None - es = twostage_fitted.aggregate("event_study") - for e, row in twostage_fit_time.event_study_effects.items(): - i = list(es.event_time).index(e) - if np.isfinite(row["effect"]) and not es.is_reference[i]: - np.testing.assert_allclose(row["effect"], es.att[i], rtol=1e-14) + params = list(inspect.signature(TwoStageDiD.fit).parameters) + assert "aggregate" not in params and "balance_e" not in params + assert params[7] == "survey_design" + d, _ = _twostage_survey_panel() + sd = _twostage_survey_design() + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + positional = TwoStageDiD().fit(d, "outcome", "unit", "period", "first_treat", None, sd) + keyword = TwoStageDiD().fit(d, survey_design=sd, **TWOSTAGE_KW) + assert positional.survey_metadata is not None + assert positional.overall_se == keyword.overall_se class TestTwoStageAggregate: @@ -4268,16 +4492,196 @@ def test_repeated_and_order_independent(self, twostage_fitted): b = twostage_fitted.aggregate("group").to_dataframe() pd.testing.assert_frame_equal(a, b) - # ---------------- bootstrapped fits (fail closed until the PR 3c relay) ---------------- + # ---------------- bootstrapped fits: the RELAY (4.0 PR 3c) ---------------- - def test_bootstrap_recompute_levels_fail_closed(self, twostage_panel): - res = _fit_twostage(twostage_panel, est_kw={"n_bootstrap": 19, "seed": 1}) + def test_bootstrap_relays_fit_time_surfaces(self, twostage_panel): + res = _fit_twostage(twostage_panel, est_kw=_TS_BOOT_KW) + assert res.bootstrap_results is not None + es, grp = res.aggregate("event_study"), res.aggregate("group") + _assert_relay_matches_stored_surfaces(res, es, grp) + orow = _ts_boot_oracle("default_all") + _assert_es_matches_oracle(es, orow, **BOOT_TOL, p_atol=_TS_P_ATOL, name="ts/boot") + _assert_group_matches_oracle(grp, orow, **BOOT_TOL, p_atol=_TS_P_ATOL, name="ts/boot") + _assert_overall_matches_oracle( + res, _ts_boot_oracle("default_plain_overall"), **BOOT_TOL, p_atol=_TS_P_ATOL + ) + _assert_overall_matches_oracle(res, orow["overall"], **BOOT_TOL, p_atol=_TS_P_ATOL) + _assert_bootstrap_p_values_in_unit_interval(es) + with pytest.raises(NotImplementedError, match="balance_e"): + res.aggregate("event_study", balance_e=1) + _assert_bootstrap_simple_relay(res) + + def test_bootstrap_pretrends_relay_matches_oracle(self, twostage_panel): + res = _fit_twostage(twostage_panel, est_kw={**_TS_BOOT_KW, "pretrends": True}) + es = res.aggregate("event_study") + _assert_es_matches_oracle( + es, _ts_boot_oracle("pretrends_all"), **BOOT_TOL, p_atol=_TS_P_ATOL, name="ts/boot/pre" + ) + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) + + def test_bootstrap_prop5_relay_matches_oracle_and_aggregator_warnings_fire(self): + """The Prop-5 panel: the relay matches the oracle (NaN rows kept with + n > 0) AND the ES aggregator's own fit-time UserWarnings now fire on + the bootstrapped PLAIN fit (documented 4.0 PR 3c change).""" + from diff_diff import TwoStageDiD + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = TwoStageDiD(**_TS_BOOT_KW).fit(_twostage_prop5_panel(), **TWOSTAGE_KW) + msgs = [str(w.message) for w in caught if issubclass(w.category, UserWarning)] + assert any("Proposition 5" in m for m in msgs) + assert not any("could not be materialized" in m for m in msgs) + assert res.bootstrap_results is not None and res.event_study_effects + orow = _ts_boot_oracle("prop5_all") + _assert_es_matches_oracle( + res.aggregate("event_study"), orow, **BOOT_TOL, p_atol=_TS_P_ATOL, name="ts/boot/p5" + ) + _assert_group_matches_oracle( + res.aggregate("group"), orow, **BOOT_TOL, p_atol=_TS_P_ATOL, name="ts/boot/p5" + ) + _assert_overall_matches_oracle( + res, _ts_boot_oracle("prop5_plain_overall"), **BOOT_TOL, p_atol=_TS_P_ATOL + ) + + def test_legacy_fields_populated_iff_bootstrapped(self, twostage_panel, twostage_fitted): + assert twostage_fitted.event_study_effects is None + assert twostage_fitted.group_effects is None + boot = _fit_twostage(twostage_panel, est_kw=_TS_BOOT_KW) + assert boot.event_study_effects and boot.group_effects + assert boot.event_study_vcov is None and boot.event_study_vcov_index is None + assert boot.event_study_df is None + + def test_pre_4_0_pickle_without_surface_raises(self, twostage_panel): + res = _fit_twostage(twostage_panel, est_kw=_TS_BOOT_KW) + legacy = dataclasses.replace(res, event_study_effects=None, group_effects=None) + with pytest.raises(ValueError, match="carries no stored event_study surface"): + legacy.aggregate("event_study") + with pytest.raises(ValueError, match="carries no stored group surface"): + legacy.aggregate("group") + _assert_bootstrap_simple_relay(legacy) + + # ---------------- fail-soft materialization (failure injection) ---------------- + + @staticmethod + def _boot_est(**extra): + from diff_diff import TwoStageDiD + + return TwoStageDiD(**_TS_BOOT_KW, **extra) + + def _assert_fail_soft_outcome(self, res): + assert res.event_study_effects is None and res.group_effects is None + assert res.event_study_vcov is None and res.event_study_vcov_index is None + assert res.bootstrap_results is not None + _assert_overall_matches_oracle( + res, _ts_boot_oracle("default_plain_overall"), **BOOT_TOL, p_atol=_TS_P_ATOL + ) for level in ("event_study", "group"): - with pytest.raises(NotImplementedError, match="bootstrap") as exc: + with pytest.raises(ValueError, match="could not be materialized"): res.aggregate(level) - assert "aggregate('simple') and, where supported, aggregate('total') relay" in str( - exc.value - ) + _assert_bootstrap_simple_relay(res) + + def test_failsoft_a_surface_hook_failure(self, twostage_panel, monkeypatch): + from diff_diff import TwoStageDiD + + def boom(self, *a, **k): + raise RuntimeError("injected surface failure") + + monkeypatch.setattr(TwoStageDiD, "_stage2_event_study", boom) + with pytest.warns(UserWarning, match="could not be materialized") as rec: + res = self._boot_est().fit(twostage_panel, **TWOSTAGE_KW) + assert any("injected surface failure" in str(w.message) for w in rec) + self._assert_fail_soft_outcome(res) + + def test_failsoft_b_engine_failure_with_surfaces_retries_overall_only( + self, twostage_panel, monkeypatch + ): + from diff_diff import TwoStageDiD + + orig = TwoStageDiD._run_bootstrap + calls = [] + + def flaky(self, *a, **k): + calls.append(k.get("original_event_study") is not None) + if k.get("original_event_study") is not None: + raise RuntimeError("injected engine failure") + return orig(self, *a, **k) + + monkeypatch.setattr(TwoStageDiD, "_run_bootstrap", flaky) + with pytest.warns(UserWarning, match="could not be materialized"): + res = self._boot_est().fit(twostage_panel, **TWOSTAGE_KW) + assert calls == [True, False] + self._assert_fail_soft_outcome(res) + + def test_failsoft_b1_lsmr_unconverged_none_return_retries(self, twostage_panel, monkeypatch): + """(b') the None-return path: `_compute_cluster_S_scores` hits + `_LSMRUnconvergedError` only on the multi-column ES/group designs + (the single-column static call succeeds), so attempt 1 RETURNS None + with surfaces present -> dropped, retried overall-only.""" + from diff_diff import TwoStageDiD + from diff_diff.two_stage import _LSMRUnconvergedError + + orig = TwoStageDiD._compute_cluster_S_scores + widths = [] + + def selective(self, *a, **k): + x2 = k["X_2"] + widths.append(int(x2.shape[1])) + if x2.shape[1] > 1: + raise _LSMRUnconvergedError("injected LSMR non-convergence") + return orig(self, *a, **k) + + monkeypatch.setattr(TwoStageDiD, "_compute_cluster_S_scores", selective) + with pytest.warns(UserWarning, match="could not be materialized") as rec: + res = self._boot_est().fit(twostage_panel, **TWOSTAGE_KW) + assert any("returned no results" in str(w.message) for w in rec) + assert widths[0] == 1 and max(widths) > 1 and widths[-1] == 1 # static, ES, retry static + self._assert_fail_soft_outcome(res) + + def test_failsoft_b2_total_engine_failure_falls_back_analytical( + self, twostage_panel, monkeypatch + ): + from diff_diff import TwoStageDiD + + def boom(self, *a, **k): + raise RuntimeError("injected total engine failure") + + monkeypatch.setattr(TwoStageDiD, "_run_bootstrap", boom) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = self._boot_est().fit(twostage_panel, **TWOSTAGE_KW) + msgs = [str(w.message) for w in caught] + assert any("Bootstrap failed" in m for m in msgs) # today's catch-all + assert not any("could not be materialized" in m for m in msgs) + assert res.bootstrap_results is None + assert res.event_study_effects is None and res.group_effects is None + analytic = _fit_twostage(twostage_panel) + assert res.overall_se == analytic.overall_se + _frame_equal( + res.aggregate("event_study").to_dataframe(), + analytic.aggregate("event_study").to_dataframe(), + ) + _frame_equal( + res.aggregate("group").to_dataframe(), analytic.aggregate("group").to_dataframe() + ) + + def test_failsoft_b2_none_return_on_both_attempts_falls_back_analytical( + self, twostage_panel, monkeypatch + ): + from diff_diff import TwoStageDiD + + monkeypatch.setattr(TwoStageDiD, "_run_bootstrap", lambda self, *a, **k: None) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = self._boot_est().fit(twostage_panel, **TWOSTAGE_KW) + assert not any("could not be materialized" in str(w.message) for w in caught) + assert res.bootstrap_results is None + assert res.event_study_effects is None and res.group_effects is None + analytic = _fit_twostage(twostage_panel) + assert res.overall_se == analytic.overall_se + _frame_equal( + res.aggregate("event_study").to_dataframe(), + analytic.aggregate("event_study").to_dataframe(), + ) def test_bootstrap_simple_relays_stored_quintet(self, twostage_panel): res = _fit_twostage(twostage_panel, est_kw={"n_bootstrap": 19, "seed": 1}) @@ -4329,7 +4733,7 @@ def test_empty_balance_window(self, twostage_fitted): # --------------------------------------------------------------------------- # -# ContinuousDiD (row M-025): fit(aggregate=) shim pin + the MIXED +# ContinuousDiD (row M-025): fit(aggregate=) removal pin + the MIXED # view/relay/recompute aggregate() - 'simple'/'dose' are pure VIEWS over stored # fields (permitted on bootstrap fits), 'event_study' is a pruned-IF-payload kit # recompute on analytic fits and a RELAY of the fit-time percentile surface on @@ -4473,12 +4877,6 @@ def cont_fitted(cont_panel): return _fit_cont(cont_panel) -@pytest.fixture(scope="module") -def cont_fit_time(cont_panel): - """Deprecated fit-time aggregate='eventstudy' - the inertness reference.""" - return _fit_cont(cont_panel, aggregate="eventstudy") - - @pytest.fixture(scope="module") def cont_bootstrap(cont_panel): return _fit_cont(cont_panel, est_kw=dict(n_bootstrap=30, seed=3)) @@ -4501,7 +4899,13 @@ def _assert_cont_overall_matches_oracle(res, orow, *, p_atol=_CO_P_ATOL): o = orow["overall"] _close(res.overall_att, o["att"], **BOOT_TOL, what="cont.overall_att") _close(res.overall_att_se, o["se"], **BOOT_TOL, what="cont.overall_att_se") - _close(res.overall_att_t_stat, o["t_stat"], **BOOT_TOL, what="cont.overall_att_t") + _close( + res.overall_att_t_stat, + o["t_stat"], + rtol=_t_rtol(BOOT_TOL["rtol"]), + atol=BOOT_TOL["atol"], + what="cont.overall_att_t", + ) _close(res.overall_att_p_value, o["p_value"], rtol=1e-13, atol=p_atol, what="cont.overall_p") _close(res.overall_att_conf_int[0], o["ci_lo"], **BOOT_TOL, what="cont.ci_lo") _close(res.overall_att_conf_int[1], o["ci_hi"], **BOOT_TOL, what="cont.ci_hi") @@ -4509,7 +4913,11 @@ def _assert_cont_overall_matches_oracle(res, orow, *, p_atol=_CO_P_ATOL): _close(res.overall_acrt_se, orow["overall_acrt_se"], **BOOT_TOL, what="cont.acrt_se") -class TestContinuousShim: +class TestContinuousRemoval: + """M-025 removed: ``fit(aggregate=)`` - including the ``"eventstudy"`` + spelling and the always-inert ``"dose"`` - is a TypeError; the 7th + positional slot is ``survey_design``.""" + def test_plain_fit_does_not_warn(self, cont_panel): from diff_diff import ContinuousDiD @@ -4518,41 +4926,29 @@ def test_plain_fit_does_not_warn(self, cont_panel): ContinuousDiD().fit(cont_panel, **CONT_KW) assert [w for w in caught if issubclass(w.category, FutureWarning)] == [] - def test_aggregate_kwarg_warns_even_at_none(self, cont_panel): - from diff_diff import ContinuousDiD - - with pytest.warns(FutureWarning, match=r"ContinuousDiD\.fit\(aggregate=\)"): - ContinuousDiD().fit(cont_panel, **CONT_KW, aggregate=None) - - def test_dose_value_warns_and_is_inert(self, cont_panel, cont_fitted): - """aggregate='dose' was always a fit-time no-op - the deprecated - call warns (the message says so) and produces the identical fit.""" + @pytest.mark.parametrize("value", [None, "dose", "eventstudy", "event_study"]) + def test_aggregate_keyword_is_a_type_error(self, cont_panel, value): from diff_diff import ContinuousDiD - with pytest.warns(FutureWarning, match="already[\\s\\n ]*redundant"): - res = ContinuousDiD().fit(cont_panel, **CONT_KW, aggregate="dose") - assert res.overall_att == cont_fitted.overall_att - assert res.overall_acrt_se == cont_fitted.overall_acrt_se - np.testing.assert_array_equal( - res.dose_response_att.effects, cont_fitted.dose_response_att.effects - ) - assert res.event_study_effects is None - - def test_warn_and_still_work(self, cont_fit_time): - """The deprecated 'eventstudy' value still computes the fit-time - surface (legacy routing unchanged until 4.0).""" - assert cont_fit_time.event_study_effects is not None - assert any(np.isfinite(v["se"]) for v in cont_fit_time.event_study_effects.values()) + with pytest.raises(TypeError, match=_unexpected("aggregate")): + ContinuousDiD().fit(cont_panel, **CONT_KW, aggregate=value) - def test_invalid_value_warns_then_raises(self, cont_panel): - """The PRE-EXISTING fit-time value validation survives the shim - (a delta vs the EfficientDiD/Imputation no-validation shims): - unknown strings still raise ValueError - after the warning.""" + def test_signature_and_seventh_positional_is_survey_design(self): from diff_diff import ContinuousDiD - with pytest.warns(FutureWarning): - with pytest.raises(ValueError, match="Invalid aggregate"): - ContinuousDiD().fit(cont_panel, **CONT_KW, aggregate="event_study") + params = list(inspect.signature(ContinuousDiD.fit).parameters) + assert "aggregate" not in params and "balance_e" not in params + assert params[7] == "survey_design" + d, _ = _cont_survey_panel() + sd = _cont_survey_design() + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + positional = ContinuousDiD().fit( + d, "outcome", "unit", "period", "first_treat", "dose", sd + ) + keyword = ContinuousDiD().fit(d, survey_design=sd, **CONT_KW) + assert positional.survey_metadata is not None + assert positional.overall_att_se == keyword.overall_att_se class TestContinuousAggregate: @@ -4834,11 +5230,30 @@ def test_mixed_type_labels_preserve_producer_order(self): assert list(frame["target"]) == ["att", "att", "acrt", "acrt"] assert "target" in agg.summary() - # ---------------- bootstrap: fail-closed gate (relay lands in PR 3c) + kit shape ---------------- + # ---------------- bootstrap: the RELAY (4.0 PR 3c) + kit shape ---------------- - def test_bootstrap_event_study_fails_closed(self, cont_bootstrap): - with pytest.raises(NotImplementedError, match="bootstrap"): - cont_bootstrap.aggregate("event_study") + def test_bootstrap_event_study_relays(self, cont_bootstrap): + assert cont_bootstrap.event_study_effects # materialized at fit + assert cont_bootstrap.event_study_df is None + es = cont_bootstrap.aggregate("event_study") + _frame_equal(es.to_dataframe(), build_event_study_surface(cont_bootstrap).to_dataframe()) + assert es.vcov is None and es.vcov_index is None + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) + stored = cont_bootstrap.event_study_effects + for i, e in enumerate(es.event_time): + row = stored[int(e)] + assert float(es.att[i]) == float(row["effect"]) + assert float(es.se[i]) == float(row["se"]) + assert float(es.p_value[i]) == float(row["p_value"]) + _assert_es_matches_oracle( + es, + _co_boot_oracle("default_eventstudy"), + **BOOT_TOL, + p_atol=_CO_P_ATOL, + name="cont/boot", + ) + _assert_bootstrap_p_values_in_unit_interval(es) + assert es.n_kind is None def test_bootstrap_plain_overall_matches_oracle(self, cont_bootstrap): """The headline bootstrap (overall ATT + ACRT) is untouched by the @@ -4861,6 +5276,88 @@ def test_bootstrap_kit_is_scalars_only(self, cont_bootstrap): assert bk["resolved_survey"] is None assert bk["n_bootstrap"] == 30 + def test_pre_4_0_pickle_without_surface_raises(self, cont_bootstrap): + legacy = dataclasses.replace(cont_bootstrap, event_study_effects=None) + with pytest.raises(ValueError, match="carries no stored event_study surface"): + legacy.aggregate("event_study") + assert legacy.aggregate("simple") is not None # views still work + + def test_bootstrap_without_post_cells_stores_no_surface(self): + """Storage-rule corner: no post-treatment cells -> the bootstrap never + runs, the surface is None and the relay raises the absent-surface + error (pre-4.0 this raised NotImplementedError from the gate).""" + from diff_diff import ContinuousDiD + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = ContinuousDiD(**_CO_BOOT_KW).fit(_cont_empty_post_panel(), **CONT_KW) + assert any("No post-treatment" in str(w.message) for w in caught) + assert not any("could not be materialized" in str(w.message) for w in caught) + assert np.isnan(res.overall_att) + assert res.event_study_effects is None + with pytest.raises(ValueError, match="carries no stored event_study surface"): + res.aggregate("event_study") + + # ---------------- fail-soft materialization (failure injection) ---------------- + + def _assert_cont_fail_soft_outcome(self, res, cont_bootstrap): + assert res.event_study_effects is None + _assert_cont_overall_matches_oracle(res, _co_boot_oracle("default_plain")) + # The dose curves equal the plain bootstrapped fit's (same seed -> same draws). + for name in ("dose_response_att", "dose_response_acrt"): + got, want = getattr(res, name), getattr(cont_bootstrap, name) + np.testing.assert_array_equal(np.asarray(got.effects), np.asarray(want.effects)) + np.testing.assert_array_equal(np.asarray(got.se), np.asarray(want.se)) + np.testing.assert_array_equal(np.asarray(got.p_value), np.asarray(want.p_value)) + with pytest.raises(ValueError, match="could not be materialized"): + res.aggregate("event_study") + assert res.aggregate("simple") is not None + + def test_failsoft_a_surface_hook_failure(self, cont_panel, cont_bootstrap, monkeypatch): + from diff_diff import ContinuousDiD + + def boom(self, *a, **k): + raise RuntimeError("injected surface failure") + + monkeypatch.setattr(ContinuousDiD, "_aggregate_event_study", boom) + with pytest.warns(UserWarning, match="could not be materialized") as rec: + res = ContinuousDiD(**_CO_BOOT_KW).fit(cont_panel, **CONT_KW) + assert any("injected surface failure" in str(w.message) for w in rec) + self._assert_cont_fail_soft_outcome(res, cont_bootstrap) + + def test_failsoft_b_engine_failure_with_surface_retries( + self, cont_panel, cont_bootstrap, monkeypatch + ): + from diff_diff import ContinuousDiD + + orig = ContinuousDiD._run_bootstrap + calls = [] + + def flaky(self, *a, **k): + es = a[12] if len(a) > 12 else k.get("event_study_effects") + calls.append(es is not None) + if es is not None: + raise RuntimeError("injected engine failure") + return orig(self, *a, **k) + + monkeypatch.setattr(ContinuousDiD, "_run_bootstrap", flaky) + with pytest.warns(UserWarning, match="could not be materialized"): + res = ContinuousDiD(**_CO_BOOT_KW).fit(cont_panel, **CONT_KW) + assert calls == [True, False] + self._assert_cont_fail_soft_outcome(res, cont_bootstrap) + + def test_failsoft_retry_failure_propagates(self, cont_panel, monkeypatch): + from diff_diff import ContinuousDiD + + def boom(self, *a, **k): + raise RuntimeError("injected total engine failure") + + monkeypatch.setattr(ContinuousDiD, "_run_bootstrap", boom) + with pytest.raises(RuntimeError, match="injected total engine failure"): + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + ContinuousDiD(**_CO_BOOT_KW).fit(cont_panel, **CONT_KW) + # ---------------- isolation (ES route only - views are views) ---------------- def test_es_isolation_from_public_field_mutation(self, cont_panel): @@ -4878,12 +5375,13 @@ def test_es_isolation_from_public_field_mutation(self, cont_panel): pd.testing.assert_frame_equal(baseline, again) def test_bootstrap_gate_reads_kit_not_field(self, cont_panel): - """The sharpest isolation arm: mutating res.n_bootstrap = 0 must - NOT bypass the fail-closed gate (it reads the kit).""" - res = _fit_cont(cont_panel, est_kw=dict(n_bootstrap=30, seed=3)) + """The sharpest isolation arm: mutating res.n_bootstrap = 0 must NOT + flip the gate (it reads the kit) - the relay still serves the stored + percentile surface instead of attempting a scalars-only recompute.""" + res = _fit_cont(cont_panel, est_kw=_CO_BOOT_KW) + before = res.aggregate("event_study").to_dataframe() res.n_bootstrap = 0 - with pytest.raises(NotImplementedError, match="bootstrap"): - res.aggregate("event_study") + _frame_equal(res.aggregate("event_study").to_dataframe(), before) def test_repeated_calls_idempotent(self, cont_fitted): a = cont_fitted.aggregate("event_study").to_dataframe() @@ -6047,7 +6545,7 @@ class TestPerRowDfProvenanceCompletion: contract: the scalar ``df_survey`` (the fit's resolved scalar inference df) persists on bootstrapped surfaces, while the per-row ``df`` column is the inference-provenance channel and is cleared under percentile - bootstrap (M-027). The container is the canonical surface on + bootstrap (M-027). Since 4.0 the container is the only surface on analytic fits and bootstrapped Imputation/ContinuousDiD fits RELAY their stored percentile surface.""" @@ -6167,13 +6665,25 @@ def test_imputation_replicate_undefined_zero_sentinel(self): assert es.df_survey == 0.0 assert np.all(np.isnan(np.asarray(es.df, dtype=float))) - def test_imputation_bootstrap_clears_at_fit_and_replay_fails_closed(self): + def test_imputation_bootstrap_clears_at_fit_and_relays(self): + """The bootstrapped Imputation fit stores its percentile surface with + a cleared df and the relay publishes a NaN df column beside the + persisted survey scalar (Row B) - and matches the oracle.""" d, _ = _imputation_survey_panel() sd = _imputation_survey_design() - boot = _fit_imputation(d, survey_design=sd, est_kw={"n_bootstrap": 30, "seed": 1}) + boot = _fit_imputation(d, survey_design=sd, est_kw=_IM_BOOT_KW) assert boot.event_study_df is None - with pytest.raises(NotImplementedError): - boot.aggregate("event_study") + assert boot.event_study_effects + es = boot.aggregate("event_study") + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) + assert es.df_survey == boot.survey_metadata.df_survey + _assert_es_matches_oracle( + es, + _im_boot_oracle("survey_tsl_all"), + **BOOT_TOL, + p_atol=_IM_P_ATOL, + name="imp/boot/tsl", + ) # ---- ContinuousDiD ------------------------------------------------ @@ -6192,7 +6702,7 @@ def test_cont_es_per_row_df_threads_survey_df(self): assert float(expected) == _co_oracle("survey_tsl_eventstudy")["event_study_df"] def test_cont_bootstrap_fit_clears_df_scalar_is_unit_level(self): - # Bootstrapped fit: _survey_df stays None -> attr None; the fit-time + # Bootstrapped fit: _survey_df stays None -> attr None; the relayed # surface's per-row df is all-NaN while the df_survey scalar reports # the (now unit-level) design df - the Row A container pin on an # implicit-PSU design. @@ -6200,14 +6710,11 @@ def test_cont_bootstrap_fit_clears_df_scalar_is_unit_level(self): from diff_diff import SurveyDesign sd = SurveyDesign(weights="w") # implicit-PSU - boot = _fit_cont( - d, - survey_design=sd, - aggregate="eventstudy", - est_kw={"n_bootstrap": 20, "seed": 3}, - ) + boot = _fit_cont(d, survey_design=sd, est_kw={"n_bootstrap": 20, "seed": 3}) assert boot.event_study_df is None - surface = build_event_study_surface(boot) + assert boot.event_study_effects # materialized at fit, relayed below + surface = boot.aggregate("event_study") + _frame_equal(surface.to_dataframe(), build_event_study_surface(boot).to_dataframe()) assert np.all(np.isnan(np.asarray(surface.df, dtype=float))) # Unit-level scalar: n_units - 1 (implicit PSU = unit), not obs-level. n_units = boot.survey_metadata.n_psu diff --git a/tests/test_continuous_did.py b/tests/test_continuous_did.py index 206c116d..cd4b59ee 100644 --- a/tests/test_continuous_did.py +++ b/tests/test_continuous_did.py @@ -20,13 +20,6 @@ from diff_diff.continuous_did_results import ContinuousDiDResults from diff_diff.prep_dgp import generate_continuous_did_data -# M-025: sites deliberately kept on the deprecated fit-time aggregate= -# (bootstrap event studies - the aggregated bootstrap surface is -# fit-time-only until replay ships - and legacy-surface pins) run under -# this module-scoped suppression; the shim behavior itself is pinned in -# tests/test_aggregate_contract.py::TestContinuousShim. -pytestmark = pytest.mark.filterwarnings(r"ignore:ContinuousDiD\.fit\(aggregate=\):FutureWarning") - # ============================================================================= # B-Spline Basis Tests # ============================================================================= @@ -372,8 +365,9 @@ def test_unbalanced_panel_same_count_different_periods(self): with pytest.raises(ValueError, match="[Uu]nbalanced"): est.fit(data, "outcome", "unit", "period", "first_treat", "dose") - def test_invalid_aggregate_raises(self): - """Invalid aggregate value should raise ValueError.""" + def test_fit_time_aggregate_kwarg_removed(self): + """4.0: the fit-time ``aggregate=`` keyword is gone (native TypeError); + the event study is a post-fit ``results.aggregate("event_study")``.""" data = pd.DataFrame( { "unit": [1, 1, 2, 2], @@ -384,19 +378,16 @@ def test_invalid_aggregate_raises(self): } ) est = ContinuousDiD() - # M-025: the shim warns on ANY supplied value BEFORE the surviving - # value validation raises - both must fire, in that order. - with pytest.warns(FutureWarning, match=r"ContinuousDiD\.fit\(aggregate=\)"): - with pytest.raises(ValueError, match="Invalid aggregate"): - est.fit( - data, - "outcome", - "unit", - "period", - "first_treat", - "dose", - aggregate="event_study", - ) + with pytest.raises(TypeError, match="unexpected keyword argument 'aggregate'"): + est.fit( + data, + "outcome", + "unit", + "period", + "first_treat", + "dose", + aggregate="event_study", + ) def test_no_never_treated_error(self): data = pd.DataFrame( @@ -507,10 +498,19 @@ def test_to_dataframe_group_time(self, results): assert "time" in df.columns assert "att_glob" in df.columns - def test_to_dataframe_event_study_error(self, results): - """Should error if event study not computed.""" - with pytest.raises(ValueError, match="[Ee]vent study"): - results.to_dataframe(level="event_study") + def test_to_dataframe_event_study_bootstrapped_populated(self, results): + """A bootstrapped fit materializes the event-study surface at fit + time (4.0: always, no fit-time kwarg), so the legacy frame is + available and agrees row-for-row with the relayed container; the + analytic-fit error is pinned in test_event_study_to_dataframe.""" + assert results.event_study_effects is not None + df = results.to_dataframe(level="event_study") + assert "relative_period" in df.columns + assert "att_glob" in df.columns + es = results.aggregate("event_study") + assert len(df) == len(es.event_time) + np.testing.assert_array_equal(df["relative_period"].to_numpy(), es.event_time) + np.testing.assert_allclose(df["att_glob"].to_numpy(), es.att, rtol=1e-14, atol=0) def test_to_dataframe_invalid_level(self, results): with pytest.raises(ValueError, match="Unknown level"): @@ -584,20 +584,16 @@ def test_event_study_computed(self): noise_sd=0.5, ) est = ContinuousDiD(n_bootstrap=49, seed=42) - results = est.fit( - data, - "outcome", - "unit", - "period", - "first_treat", - "dose", - aggregate="eventstudy", - ) + results = est.fit(data, "outcome", "unit", "period", "first_treat", "dose") + # bootstrapped fits materialize the event-study surface at fit time + # and the post-fit container relays it assert results.event_study_effects is not None + es = results.aggregate("event_study") # Should have pre and post relative periods rel_periods = sorted(results.event_study_effects.keys()) assert min(rel_periods) < 0 # Pre-treatment assert max(rel_periods) >= 0 # Post-treatment + assert sorted(int(e) for e in es.event_time) == rel_periods def test_event_study_to_dataframe(self): data = generate_continuous_did_data( @@ -669,15 +665,8 @@ def test_event_study_not_yet_treated_bootstrap(self, ci_params): n_bootstrap=n_boot, seed=42, ) - results = est.fit( - data, - "outcome", - "unit", - "period", - "first_treat", - "dose", - aggregate="eventstudy", - ) + results = est.fit(data, "outcome", "unit", "period", "first_treat", "dose") + # bootstrapped fits populate the surface at fit time assert results.event_study_effects is not None rel_periods = sorted(results.event_study_effects.keys()) assert min(rel_periods) < 0 # Pre-treatment diff --git a/tests/test_diagnostic_report.py b/tests/test_diagnostic_report.py index b65bc9b5..8f73f7ff 100644 --- a/tests/test_diagnostic_report.py +++ b/tests/test_diagnostic_report.py @@ -109,7 +109,7 @@ def cs_fit(): # methodology-critical reason (Rambachan-Roth bounds are not valid for # interpretation on consecutive-comparison pre-periods). See the # round-5 CI review on PR #318. - # Plain fit, no fit-time surface - DR derives the event study via + # 4.0: no fit-time surface - DR derives the event study via # results.aggregate("event_study") (the default route). cs = CallawaySantAnna(base_period="universal").fit( sdf, @@ -401,7 +401,7 @@ def test_precomputed_pretrends_power_parity_with_default_path(self, cs_fit): fit, data = cs_fit # Precompute the power result from the same fit's event-study - # container (the consumer takes the post-fit container). The + # container (4.0: the consumer takes the post-fit container). The # compute function records ``covariance_source`` and populates # ``original_results`` on the output for DR's precomputed adapter. es = fit.aggregate("event_study") @@ -3507,7 +3507,7 @@ def imp_pretrends_fits(): ``pretrends=True`` is REQUIRED for estimated pre-period horizons to exist at all — under the default ``pretrends=False`` the derived - surface carries only the reference lead. A plain analytic fit + surface carries only the reference lead. Since 4.0 an analytic fit stores no surface, so the raw-field route is a legacy stand-in with the container's surface hand-set in the 3.x fields. """ @@ -4200,22 +4200,79 @@ def warning_aggregate(self, *args, **kwargs): assert any("single-copy caveat" in w for w in d[check].get("warnings", [])) assert sum("single-copy caveat" in w for w in d["warnings"]) == 1 - def test_bootstrap_failure_heterogeneity_skips_with_context(self): - # Derived heterogeneity producers on a bootstrapped fit: the runner - # section fails closed with the derivation context and publishes no - # numeric heterogeneity fields. - from diff_diff import ImputationDiD + @staticmethod + def _bootstrapped_relay_fits(): + from diff_diff import ImputationDiD, TwoStageDiD sdf = generate_staggered_data(n_units=100, n_periods=6, treatment_effect=1.5, seed=7) + kw = dict(outcome="outcome", unit="unit", time="period", first_treat="first_treat") with warnings.catch_warnings(): warnings.simplefilter("ignore") - imp = ImputationDiD(pretrends=True, n_bootstrap=19, seed=1).fit( - sdf, outcome="outcome", unit="unit", time="period", first_treat="first_treat" - ) - d = DiagnosticReport(imp).run_all().to_dict() + imp = ImputationDiD(pretrends=True, n_bootstrap=19, seed=1).fit(sdf, **kw) + ts = TwoStageDiD(pretrends=True, n_bootstrap=19, seed=1).fit(sdf, **kw) + return imp, ts + + def test_bootstrap_relay_heterogeneity_runs_from_group_effects(self): + # 4.0: a bootstrapped plain ImputationDiD / TwoStageDiD fit STORES the + # percentile event-study AND group surfaces (aggregate() relays them), + # so DR heterogeneity RUNS - sourced from the stored ``group_effects`` + # (``_collect_effect_scalars`` reads group_effects first) - and + # publishes numeric fields that match the relayed group container. + for res in self._bootstrapped_relay_fits(): + assert res.bootstrap_results is not None + assert res.event_study_effects is not None + assert res.group_effects is not None + het = DiagnosticReport(res).run_all().to_dict()["heterogeneity"] + assert het["status"] == "ran", het + assert het["source"] == "group_effects" + grp = res.aggregate("group") + finite = grp.att[np.isfinite(grp.att)] + assert het["n_effects"] == finite.size > 0 + np.testing.assert_allclose(het["mean"], float(np.mean(finite)), rtol=1e-12) + np.testing.assert_allclose(het["min"], float(np.min(finite)), rtol=1e-12) + np.testing.assert_allclose(het["max"], float(np.max(finite)), rtol=1e-12) + assert het["sign_consistent"] == bool(np.all(finite >= 0) or np.all(finite <= 0)) + + def test_bootstrap_relay_parallel_trends_runs_on_relayed_surface(self): + # The percentile event-study surface (pretrends=True leads) the + # bootstrapped fit STORES - and aggregate("event_study") relays - + # feeds DR's parallel-trends check: it RUNS. DR reads the stored raw + # field first, so this is the raw-field route (no derived-route + # provenance key), and the field equals the relayed container. + for res in self._bootstrapped_relay_fits(): + es = res.aggregate("event_study") + assert es.vcov is None # percentile surface: no covariance + assert ((es.event_time < -1) & np.isfinite(es.p_value)).any() + times = es.event_time.tolist() + for e, row in res.event_study_effects.items(): + assert row["effect"] == es.att[times.index(e)] + pt = DiagnosticReport(res).run_all().to_dict()["parallel_trends"] + assert pt["status"] == "ran", pt + assert "pre_period_source" not in pt + # No covariance -> the per-row Bonferroni path, never a joint Wald. + assert pt["method"] == "bonferroni" + assert np.isfinite(pt["joint_p_value"]) + assert {p["period"] for p in pt["per_period"]} == { + int(e) for e, p in zip(es.event_time, es.p_value) if e < -1 and np.isfinite(p) + } + + def test_bootstrap_absent_surface_heterogeneity_skips_with_context(self): + # A bootstrapped fit WITHOUT the stored surfaces (a pre-4.0 pickle, + # or a fit whose fail-soft materialization dropped them) fails + # closed: BOTH raw fields cleared (the collector reads group_effects + # first), the derivation hits the relay's absent-surface ValueError, + # and the section publishes the context and no numeric fields. + import dataclasses + + imp, _ = self._bootstrapped_relay_fits() + stripped = dataclasses.replace(imp, event_study_effects=None, group_effects=None) + with pytest.raises(ValueError, match="no stored event_study surface"): + stripped.aggregate("event_study") + d = DiagnosticReport(stripped).run_all().to_dict() het = d["heterogeneity"] assert het["status"] == "skipped" assert "aggregate('event_study')" in het["reason"] - assert "NotImplementedError" in het["reason"] + assert "ValueError" in het["reason"] + assert "no stored event_study surface" in het["reason"] assert "n_effects" not in het assert "mean" not in het diff --git a/tests/test_event_study_surface.py b/tests/test_event_study_surface.py index ac9cf7b1..cb568856 100644 --- a/tests/test_event_study_surface.py +++ b/tests/test_event_study_surface.py @@ -266,15 +266,14 @@ class _Fake: def surfaces(): """Build every producer's surface once; return {label: (native, surface)}. - Producer routes as they stand after 4.0 PR 3b: the CallawaySantAnna and - EfficientDiD "native" sides are ``legacy_standin``s carrying - ``aggregate("event_study")`` in the 3.x fields (neither fit stores a - fit-time dict any more - the keyword is gone; the stand-in is the - legacy-pickle shape ``build_event_study_surface`` still accepts), while the - ImputationDiD, TwoStageDiD and ContinuousDiD producers still use the - deprecated fit-time ``aggregate=`` route, which populates the dict directly. - Those sites move to stand-ins / the post-fit relay when their fit-time - keywords are removed (4.0 PR 3c). + Since 4.0 no CallawaySantAnna / EfficientDiD fit, and no ANALYTIC + ImputationDiD / TwoStageDiD / ContinuousDiD fit, stores a fit-time + event-study dict, so for those producers the "native" side is a legacy + stand-in carrying ``aggregate("event_study")`` in the 3.x fields (the + legacy-pickle shape ``build_event_study_surface`` still accepts). The + BOOTSTRAPPED ImputationDiD / TwoStageDiD / ContinuousDiD fits populate the + dict themselves (the percentile surface the relay serves) and enter as + genuine native producers. """ out = {} @@ -316,7 +315,7 @@ def surfaces(): time="period", first_treat="first_treat", ) - assert cs.event_study_effects is None # plain fit: not populated at fit + assert cs.event_study_effects is None # 4.0: never populated at fit cs = legacy_standin(cs) out["CallawaySantAnna"] = (cs, build_event_study_surface(cs)) @@ -335,10 +334,23 @@ def surfaces(): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) + assert imp.event_study_effects is None # analytic fit: no surface stored + imp = legacy_standin(imp) out["ImputationDiD"] = (imp, build_event_study_surface(imp)) + # 3b. Bootstrapped ImputationDiD: a GENUINE native producer (the fit + # stores the percentile surface the relay serves). + imp_boot = ImputationDiD(n_bootstrap=49, seed=42).fit( + imp_data, + outcome="outcome", + unit="unit", + time="time", + first_treat="first_treat", + ) + assert imp_boot.event_study_effects is not None + out["ImputationDiD_bootstrap"] = (imp_boot, build_event_study_surface(imp_boot)) + # 4. TwoStageDiD (same panel shape) ts = TwoStageDiD().fit( imp_data, @@ -346,10 +358,22 @@ def surfaces(): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) + assert ts.event_study_effects is None + ts = legacy_standin(ts) out["TwoStageDiD"] = (ts, build_event_study_surface(ts)) + # 4b. Bootstrapped TwoStageDiD: genuine native producer. + ts_boot = TwoStageDiD(n_bootstrap=49, seed=42).fit( + imp_data, + outcome="outcome", + unit="unit", + time="time", + first_treat="first_treat", + ) + assert ts_boot.event_study_effects is not None + out["TwoStageDiD_bootstrap"] = (ts_boot, build_event_study_surface(ts_boot)) + # 5. StackedDiD st_data = dgp_staggered( n_units=120, @@ -392,14 +416,23 @@ def surfaces(): n_units=200, n_periods=5, cohort_periods=[2, 4], seed=42, noise_sd=0.5 ) cd = ContinuousDiD(control_group="not_yet_treated", n_bootstrap=0).fit( - cd_data, "outcome", "unit", "period", "first_treat", "dose", aggregate="eventstudy" + cd_data, "outcome", "unit", "period", "first_treat", "dose" ) + assert cd.event_study_effects is None + cd = legacy_standin(cd) out["ContinuousDiD"] = (cd, build_event_study_surface(cd)) + # 7b. Bootstrapped ContinuousDiD: genuine native producer. + cd_boot = ContinuousDiD(control_group="not_yet_treated", n_bootstrap=49, seed=42).fit( + cd_data, "outcome", "unit", "period", "first_treat", "dose" + ) + assert cd_boot.event_study_effects is not None + out["ContinuousDiD_bootstrap"] = (cd_boot, build_event_study_surface(cd_boot)) + # 8. EfficientDiD ed_data = _efficient_panel() ed = EfficientDiD().fit(ed_data, "y", "unit", "time", "first_treat") - assert ed.event_study_effects is None # 4.0 (PR 3b): never populated at fit + assert ed.event_study_effects is None ed = legacy_standin(ed) out["EfficientDiD"] = (ed, build_event_study_surface(ed)) @@ -468,10 +501,13 @@ def surfaces(): "CallawaySantAnna", "SunAbraham", "ImputationDiD", + "ImputationDiD_bootstrap", "TwoStageDiD", + "TwoStageDiD_bootstrap", "StackedDiD", "SpilloverDiD", "ContinuousDiD", + "ContinuousDiD_bootstrap", "EfficientDiD", "WooldridgeDiD", "StaggeredTripleDifference", @@ -1044,7 +1080,7 @@ def test_absent_surface_raises(): from diff_diff.prep import generate_staggered_data as prep_staggered data = prep_staggered(n_units=60, n_periods=6, cohort_periods=[3], treatment_effect=2.0, seed=1) - # A plain CallawaySantAnna fit does not populate event_study_effects; the builder + # 4.0: no CallawaySantAnna fit populates event_study_effects; the builder # distinguishes that ABSENT surface (raise) from an empty one. cs = CallawaySantAnna(n_bootstrap=0).fit( data, outcome="outcome", unit="unit", time="period", first_treat="first_treat" @@ -1323,6 +1359,19 @@ def test_two_stage_vcov_alignment(surfaces): assert np.isnan(diag[i]) # rank-guard NaN row == NaN marginal SE +@pytest.mark.parametrize( + "label", ["ImputationDiD_bootstrap", "TwoStageDiD_bootstrap", "ContinuousDiD_bootstrap"] +) +def test_bootstrap_relay_native_surface_has_no_vcov(label, surfaces): + # The bootstrapped adopters store the percentile surface; its SEs are + # not the diagonal of any stored matrix, so the surface carries no vcov + # and the relay container is the same surface. + native, surface = surfaces[label] + assert surface.vcov is None and surface.vcov_index is None + relay = native.aggregate("event_study") + pd.testing.assert_frame_equal(relay.to_dataframe(), surface.to_dataframe()) + + def test_two_stage_prop5_horizons_excluded_from_vcov_index(): # No never-treated units -> late horizons are Proposition-5 unidentified # (NaN effect, n_obs > 0). They live in the effects dict but were never diff --git a/tests/test_guides.py b/tests/test_guides.py index cb588db1..fd85bb88 100644 --- a/tests/test_guides.py +++ b/tests/test_guides.py @@ -297,9 +297,8 @@ def test_module_docstring_mentions_helper(): # llms-full.txt — HeterogeneousAdoptionDiD coverage (Phase 5) # --------------------------------------------------------------------------- def _assert_no_fit_time_aggregation_lines(fit_block: str, label: str) -> None: - """4.0 (PR 3b): a documented fit() signature whose fit-time ``aggregate`` / - ``balance_e`` keywords were removed carries no such line - aggregation is - post-fit only there.""" + """4.0 (PRs 3b/3c): no documented fit() signature carries a fit-time + ``aggregate`` / ``balance_e`` line - aggregation is post-fit only.""" offending = [ line for line in fit_block.splitlines() @@ -1041,42 +1040,25 @@ def _section(self, header): nxt = text.index("\n### ", start + 1) return text[start:nxt] - def _assert_shim_lines(self, section, fit_call, agg_row, bal_row=None): - """``bal_row=None`` skips the balance_e-line lookup - required for - estimators without a balance_e param (ContinuousDiD, M-025), where - the unconditional ``next(...)`` would raise StopIteration.""" - fit_start = section.index(fit_call) - fit_block = section[fit_start : section.index("\n)", fit_start)] - agg_line = next( - line for line in fit_block.splitlines() if line.strip().startswith("aggregate") - ) - assert "NOT_SUPPLIED" in agg_line - assert f"DEPRECATED ({agg_row})" in agg_line - assert "results.aggregate()" in agg_line - if bal_row is not None: - bal_line = next( - line for line in fit_block.splitlines() if line.strip().startswith("balance_e") - ) - assert "NOT_SUPPLIED" in bal_line - assert f"DEPRECATED ({bal_row})" in bal_line - def _assert_no_shim_lines(self, section, fit_call): fit_start = section.index(fit_call) fit_block = section[fit_start : section.index("\n)", fit_start)] _assert_no_fit_time_aggregation_lines(fit_block, fit_call) assert "NOT_SUPPLIED" not in fit_block - def test_llms_full_imputation_fit_aggregate_line_documents_shim(self): - # The M-021/M-118 twin of the EfficientDiD pin above - no other - # test covers the ImputationDiD signature block's shim comments. - self._assert_shim_lines(self._section("### ImputationDiD"), "imp.fit(", "M-021", "M-118") + def test_llms_full_imputation_fit_block_has_no_fit_time_aggregate(self): + # M-021 / M-118 (removed at 4.0, PR 3c): the M-021/M-118 twin of the + # EfficientDiD pin above - no other test covers the ImputationDiD + # signature block. + self._assert_no_shim_lines(self._section("### ImputationDiD"), "imp.fit(") - def test_llms_full_two_stage_fit_aggregate_line_documents_shim(self): - self._assert_shim_lines(self._section("### TwoStageDiD"), ".fit(", "M-022", "M-119") + def test_llms_full_two_stage_fit_block_has_no_fit_time_aggregate(self): + # M-022 / M-119 (removed at 4.0, PR 3c). + self._assert_no_shim_lines(self._section("### TwoStageDiD"), ".fit(") - def test_llms_full_continuous_fit_aggregate_line_documents_shim(self): - # M-025: no balance_e twin exists - bal_row stays None. - self._assert_shim_lines(self._section("### ContinuousDiD"), "cdid.fit(", "M-025") + def test_llms_full_continuous_fit_block_has_no_fit_time_aggregate(self): + # M-025 (removed at 4.0, PR 3c; no balance_e twin exists). + self._assert_no_shim_lines(self._section("### ContinuousDiD"), "cdid.fit(") def test_llms_full_had_fit_block_has_no_fit_time_aggregate(self): # M-027 (removed at 4.0, PR 3b): the mode selector is gone from the diff --git a/tests/test_imputation.py b/tests/test_imputation.py index 4377daf2..60e8398e 100644 --- a/tests/test_imputation.py +++ b/tests/test_imputation.py @@ -16,21 +16,6 @@ ) from diff_diff.survey import SurveyDesign -# --------------------------------------------------------------------------- -# Rows M-021/M-022 (+ M-118/M-119): ImputationDiD / TwoStageDiD -# ``fit(aggregate=, balance_e=)`` is deprecated (3.9, removed 4.0) and warns on -# ANY supplied value. The deprecated fit-time route is kept DELIBERATELY here: -# these tests pin FIT-TIME surface behaviour (bit-equality grids, bootstrap -# aggregation, R/Stata parity, replicate overrides, native effect dicts) that -# the post-fit ``results.aggregate(...)`` container route does not reproduce -# shape-for-shape. The shim warning is therefore filtered BY MESSAGE, scoped to -# these two estimators only - every other FutureWarning (including the other -# estimators' aggregate() shims) still surfaces. -# --------------------------------------------------------------------------- -pytestmark = pytest.mark.filterwarnings( - r"ignore:(ImputationDiD|TwoStageDiD)\.fit\((aggregate=|balance_e=|aggregate= / balance_e=)\):FutureWarning" -) - # ============================================================================= # Shared test data generation # ============================================================================= @@ -412,7 +397,7 @@ def test_to_dataframe_errors(self): with pytest.raises(ValueError, match="Unknown level"): results.to_dataframe("invalid") - with pytest.raises(ValueError, match="Event study effects not computed"): + with pytest.raises(ValueError, match="Event study effects not stored on this fit"): results.to_dataframe("event_study") def test_get_params(self): @@ -961,7 +946,8 @@ def test_bootstrap_different_seeds(self, ci_params): assert r1.overall_se != r2.overall_se def test_bootstrap_event_study(self, ci_params): - """Test bootstrap with event study aggregation.""" + """A bootstrapped fit materializes the percentile event-study surface + at fit and ``aggregate('event_study')`` relays it.""" data = generate_test_data() n_boot = ci_params.bootstrap(50) est = ImputationDiD(n_bootstrap=n_boot, seed=42) @@ -971,15 +957,23 @@ def test_bootstrap_event_study(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) br = results.bootstrap_results assert br.event_study_ses is not None assert len(br.event_study_ses) > 0 + assert results.event_study_effects is not None + relayed = es_dict(results) + assert set(relayed) == set(results.event_study_effects) + for h, eff in results.event_study_effects.items(): + if eff["n_obs"] == 0: + continue # reference row: the container reports NaN inference + assert relayed[h]["effect"] == eff["effect"] + assert relayed[h]["se"] == eff["se"] def test_bootstrap_group(self, ci_params): - """Test bootstrap with group aggregation.""" + """A bootstrapped fit materializes the percentile group surface at + fit and ``aggregate('group')`` relays it.""" data = generate_test_data() n_boot = ci_params.bootstrap(50) est = ImputationDiD(n_bootstrap=n_boot, seed=42) @@ -989,62 +983,48 @@ def test_bootstrap_group(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="group", ) br = results.bootstrap_results assert br.group_ses is not None assert len(br.group_ses) == 3 - - def test_bootstrap_balance_e_consistency(self, ci_params): - """Test bootstrap event study respects balance_e filtering.""" + assert results.group_effects is not None + relayed = group_dict(results) + assert set(relayed) == set(results.group_effects) + for g, eff in results.group_effects.items(): + assert relayed[g]["effect"] == eff["effect"] + assert relayed[g]["se"] == eff["se"] + + def test_bootstrap_balance_e_rejected_post_fit(self, ci_params): + """The fit-time percentile surface is computed with balance_e=None + and the per-target draws are not retained, so ``balance_e`` on a + bootstrapped fit is rejected post-fit (the unrestricted relay stays + available).""" data = generate_test_data(n_units=150, seed=42) n_boot = ci_params.bootstrap(50) - # Run WITH balance_e - est_bal = ImputationDiD(n_bootstrap=n_boot, seed=42) - results_bal = est_bal.fit( - data, - outcome="outcome", - unit="unit", - time="time", - first_treat="first_treat", - aggregate="event_study", - balance_e=2, - ) - - # Run WITHOUT balance_e - est_nobal = ImputationDiD(n_bootstrap=n_boot, seed=42) - results_nobal = est_nobal.fit( + est = ImputationDiD(n_bootstrap=n_boot, seed=42) + results = est.fit( data, outcome="outcome", unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) - assert results_bal.bootstrap_results is not None - assert results_bal.bootstrap_results.event_study_ses is not None + assert results.bootstrap_results is not None + assert results.bootstrap_results.event_study_ses is not None - # Verify SEs are finite - for h in results_bal.event_study_effects: - eff = results_bal.event_study_effects[h] - if eff.get("n_obs", 0) > 0 and np.isfinite(eff["effect"]): - if h in results_bal.bootstrap_results.event_study_ses: - assert np.isfinite(results_bal.bootstrap_results.event_study_ses[h]) + with pytest.raises(NotImplementedError, match="balance_e"): + results.aggregate("event_study", balance_e=2) - # Verify balance_e changed bootstrap SEs at some horizon - if results_nobal.bootstrap_results is not None: - bal_ses = results_bal.bootstrap_results.event_study_ses - nobal_ses = results_nobal.bootstrap_results.event_study_ses - shared_h = set(bal_ses.keys()) & set(nobal_ses.keys()) - any_different = any( - not np.isclose(bal_ses[h], nobal_ses[h], rtol=0.05) - for h in shared_h - if np.isfinite(bal_ses[h]) and np.isfinite(nobal_ses[h]) - ) - assert any_different, "balance_e should change bootstrap SEs for at least one horizon" + # The unrestricted relay still works and its SEs are the bootstrap SEs. + relayed = es_dict(results) + for h, eff in relayed.items(): + if eff["n_obs"] > 0 and np.isfinite(eff["effect"]): + if h in results.bootstrap_results.event_study_ses: + assert np.isfinite(results.bootstrap_results.event_study_ses[h]) + assert eff["se"] == results.bootstrap_results.event_study_ses[h] def test_bootstrap_p_value_significance(self, ci_params): """Test bootstrap p-value for significant effect.""" @@ -1159,7 +1139,6 @@ def test_bootstrap_weights_event_study(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) br = results.bootstrap_results @@ -1181,7 +1160,6 @@ def test_bootstrap_weights_group(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="group", ) br = results.bootstrap_results @@ -2365,8 +2343,17 @@ class TestImputationDiDVcovType: # ---- Surface 1: default bit-equal across aggregation modes ------------ - @pytest.mark.parametrize("aggregate", [None, "event_study", "group"]) - def test_default_hc1_bit_equal_baseline(self, aggregate): + @staticmethod + def _assert_levels_bit_equal(r_default, r_explicit): + """Post-fit event-study and group containers agree bit-for-bit.""" + for level in ("event_study", "group"): + pd.testing.assert_frame_equal( + r_default.aggregate(level).to_dataframe(), + r_explicit.aggregate(level).to_dataframe(), + check_exact=True, + ) + + def test_default_hc1_bit_equal_baseline(self): data = generate_test_data(seed=53, n_units=80, n_periods=8) common = dict( data=data, @@ -2374,17 +2361,16 @@ def test_default_hc1_bit_equal_baseline(self, aggregate): unit="unit", time="time", first_treat="first_treat", - aggregate=aggregate, ) r_default = ImputationDiD().fit(**common) r_explicit = ImputationDiD(vcov_type="hc1").fit(**common) assert r_default.overall_att == r_explicit.overall_att assert r_default.overall_se == r_explicit.overall_se + self._assert_levels_bit_equal(r_default, r_explicit) # ---- Surface 2: cluster path bit-equal -------------------------------- - @pytest.mark.parametrize("aggregate", [None, "event_study", "group"]) - def test_cluster_hc1_bit_equal_baseline(self, aggregate): + def test_cluster_hc1_bit_equal_baseline(self): data = _imputation_clustered_panel() common = dict( data=data, @@ -2392,17 +2378,16 @@ def test_cluster_hc1_bit_equal_baseline(self, aggregate): unit="unit", time="time", first_treat="first_treat", - aggregate=aggregate, ) r_default = ImputationDiD(cluster="state").fit(**common) r_explicit = ImputationDiD(cluster="state", vcov_type="hc1").fit(**common) assert r_default.overall_att == r_explicit.overall_att assert r_default.overall_se == r_explicit.overall_se + self._assert_levels_bit_equal(r_default, r_explicit) # ---- Surface 3: TSL-survey path bit-equal ----------------------------- - @pytest.mark.parametrize("aggregate", [None, "event_study", "group"]) - def test_survey_tsl_hc1_bit_equal_baseline(self, aggregate): + def test_survey_tsl_hc1_bit_equal_baseline(self): data = _imputation_survey_panel() design = SurveyDesign( weights="weight", @@ -2416,18 +2401,18 @@ def test_survey_tsl_hc1_bit_equal_baseline(self, aggregate): unit="unit", time="time", first_treat="first_treat", - aggregate=aggregate, survey_design=design, ) r_default = ImputationDiD().fit(**common) r_explicit = ImputationDiD(vcov_type="hc1").fit(**common) assert r_default.overall_att == r_explicit.overall_att assert r_default.overall_se == r_explicit.overall_se + self._assert_levels_bit_equal(r_default, r_explicit) # ---- Surface 4: replicate-survey path bit-equal ----------------------- - @pytest.mark.parametrize("aggregate", [None, "event_study", "group"]) - def test_survey_replicate_hc1_bit_equal_baseline(self, aggregate): + @pytest.mark.parametrize("level", ["event_study", "group"]) + def test_survey_replicate_hc1_bit_equal_baseline(self, level): data, rep_cols = _imputation_replicate_panel() design = SurveyDesign( weights="weight", @@ -2442,32 +2427,36 @@ def test_survey_replicate_hc1_bit_equal_baseline(self, aggregate): time="time", first_treat="first_treat", survey_design=design, - aggregate=aggregate, ) r_default = ImputationDiD().fit(**common) r_explicit = ImputationDiD(vcov_type="hc1").fit(**common) assert r_default.overall_att == r_explicit.overall_att assert r_default.overall_se == r_explicit.overall_se # Per-horizon / per-group SE override branches must also agree under - # the replicate-weight variance path. - if aggregate == "event_study": - assert r_default.event_study_effects is not None - assert r_explicit.event_study_effects is not None - for h in r_default.event_study_effects: - assert ( - r_default.event_study_effects[h]["se"] - == r_explicit.event_study_effects[h]["se"] - ) - if aggregate == "group": - assert r_default.group_effects is not None - assert r_explicit.group_effects is not None - for g in r_default.group_effects: - assert r_default.group_effects[g]["se"] == r_explicit.group_effects[g]["se"] + # the replicate-weight variance path (post-fit replicate replay). + a_default = r_default.aggregate(level) + a_explicit = r_explicit.aggregate(level) + assert len(a_default.se) > 0 + assert np.array_equal(a_default.se, a_explicit.se, equal_nan=True) + assert np.array_equal(a_default.att, a_explicit.att, equal_nan=True) # ---- Surface 5: bootstrap × cluster / × survey bit-equal -------------- - @pytest.mark.parametrize("aggregate", [None, "event_study", "group"]) - def test_bootstrap_cluster_hc1_bit_equal(self, ci_params, aggregate): + @staticmethod + def _assert_bootstrap_surfaces_bit_equal(r_default, r_explicit): + """Per-horizon / per-group bootstrap SE override branches at + imputation_aggregation.py::_replicate_override_aggregates must agree + (bootstrapped fits always materialize both surfaces).""" + assert r_default.bootstrap_results.event_study_ses is not None + assert r_explicit.bootstrap_results.event_study_ses is not None + for h, se in r_default.bootstrap_results.event_study_ses.items(): + assert se == r_explicit.bootstrap_results.event_study_ses[h] + assert r_default.bootstrap_results.group_ses is not None + assert r_explicit.bootstrap_results.group_ses is not None + for g, se in r_default.bootstrap_results.group_ses.items(): + assert se == r_explicit.bootstrap_results.group_ses[g] + + def test_bootstrap_cluster_hc1_bit_equal(self, ci_params): data = _imputation_clustered_panel() n_boot = ci_params.bootstrap(199) common = dict( @@ -2476,7 +2465,6 @@ def test_bootstrap_cluster_hc1_bit_equal(self, ci_params, aggregate): unit="unit", time="time", first_treat="first_treat", - aggregate=aggregate, ) r_default = ImputationDiD(cluster="state", n_bootstrap=n_boot, seed=11).fit(**common) r_explicit = ImputationDiD( @@ -2488,21 +2476,9 @@ def test_bootstrap_cluster_hc1_bit_equal(self, ci_params, aggregate): r_default.bootstrap_results.overall_att_se == r_explicit.bootstrap_results.overall_att_se ) - # Per-horizon / per-group bootstrap SE override branches at - # imputation_aggregation.py::_replicate_override_aggregates must also agree. - if aggregate == "event_study": - assert r_default.bootstrap_results.event_study_ses is not None - assert r_explicit.bootstrap_results.event_study_ses is not None - for h, se in r_default.bootstrap_results.event_study_ses.items(): - assert se == r_explicit.bootstrap_results.event_study_ses[h] - if aggregate == "group": - assert r_default.bootstrap_results.group_ses is not None - assert r_explicit.bootstrap_results.group_ses is not None - for g, se in r_default.bootstrap_results.group_ses.items(): - assert se == r_explicit.bootstrap_results.group_ses[g] - - @pytest.mark.parametrize("aggregate", [None, "event_study", "group"]) - def test_bootstrap_survey_hc1_bit_equal(self, ci_params, aggregate): + self._assert_bootstrap_surfaces_bit_equal(r_default, r_explicit) + + def test_bootstrap_survey_hc1_bit_equal(self, ci_params): data = _imputation_survey_panel() design = SurveyDesign( weights="weight", @@ -2518,7 +2494,6 @@ def test_bootstrap_survey_hc1_bit_equal(self, ci_params, aggregate): time="time", first_treat="first_treat", survey_design=design, - aggregate=aggregate, ) r_default = ImputationDiD(n_bootstrap=n_boot, seed=23).fit(**common) r_explicit = ImputationDiD(n_bootstrap=n_boot, seed=23, vcov_type="hc1").fit(**common) @@ -2528,16 +2503,7 @@ def test_bootstrap_survey_hc1_bit_equal(self, ci_params, aggregate): r_default.bootstrap_results.overall_att_se == r_explicit.bootstrap_results.overall_att_se ) - if aggregate == "event_study": - assert r_default.bootstrap_results.event_study_ses is not None - assert r_explicit.bootstrap_results.event_study_ses is not None - for h, se in r_default.bootstrap_results.event_study_ses.items(): - assert se == r_explicit.bootstrap_results.event_study_ses[h] - if aggregate == "group": - assert r_default.bootstrap_results.group_ses is not None - assert r_explicit.bootstrap_results.group_ses is not None - for g, se in r_default.bootstrap_results.group_ses.items(): - assert se == r_explicit.bootstrap_results.group_ses[g] + self._assert_bootstrap_surfaces_bit_equal(r_default, r_explicit) # ---- Surface 6: eager transactional validation (BaseEstimator) -------- @@ -3312,46 +3278,43 @@ def test_survey_df_precedence_on_leads(self): assert e["conf_int"] == e0["conf_int"] def test_inert_config_warns_on_explicit_nondefault(self): - """M-127 REACHABILITY predicate (revised with the M-021 post-fit - migration): warn iff the per-lead inference is unreachable on every - route for this fit config — pretrends=False always warns; - pretrends=True analytical fits never warn (post-fit - results.aggregate('event_study') reaches the leads regardless of - the deprecated fit-time aggregate value); pretrends=True with - n_bootstrap>0 and aggregate unset warns (no ES surface is built - and post-fit aggregate() fails closed); pretrends=True with the - deprecated fit-time ES supplied never warns, bootstrap included - (fit-time leads use analytical inference).""" + """M-127 REACHABILITY predicate (4.0, fit-time aggregate= removed): + the per-lead inference is reachable iff ``pretrends and not + replicate`` - analytical fits recompute the leads in + results.aggregate('event_study'), bootstrapped fits materialize the + lead surface at fit and relay it; under a replicate-weight design + the post-fit route fails closed. So warn iff pretrends=False, or + pretrends=True with a replicate-weight design.""" data = self._panel() base = dict(outcome="outcome", unit="unit", time="time", first_treat="first_treat") with pytest.warns(UserWarning, match="affects only the pretrends"): ImputationDiD(df_convention="cluster").fit(data, **base) - # pretrends=True + deprecated aggregate='group': the knob is NO - # LONGER inert (post-fit ES reaches the leads) — no inert-config - # warning (the FutureWarning from aggregate= is separate). + # pretrends=True, analytical: reachable post-fit -> no warning. with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") - ImputationDiD(pretrends=True, df_convention="cluster").fit( - data, aggregate="group", **base - ) + ImputationDiD(pretrends=True, df_convention="cluster").fit(data, **base) assert not any("affects only the pretrends" in str(w.message) for w in caught) + # pretrends=True + bootstrap: the lead surface is materialized at + # fit (analytical lead inference) and relayed -> no warning. with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") - ImputationDiD(pretrends=True, df_convention="cluster").fit(data, **self._kw) - assert not any("affects only the pretrends" in str(w.message) for w in caught) - # pretrends=True + bootstrap, aggregate unset: unreachable → warns. - with pytest.warns(UserWarning, match="affects only the pretrends"): ImputationDiD(pretrends=True, df_convention="cluster", n_bootstrap=9, seed=1).fit( data, **base ) - # pretrends=True + bootstrap + deprecated fit-time ES: reachable - # (analytical lead inference rides the fit-time surface) → no warn. - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - ImputationDiD(pretrends=True, df_convention="cluster", n_bootstrap=9, seed=1).fit( - data, aggregate="event_study", **base - ) assert not any("affects only the pretrends" in str(w.message) for w in caught) + # pretrends=True + replicate-weight design: the post-fit lead route + # fails closed (no per-replicate lead refits) -> unreachable -> warns. + rep_data, rep_cols = _imputation_replicate_panel() + design = SurveyDesign( + weights="weight", + replicate_weights=rep_cols, + replicate_method="JK1", + weight_type="pweight", + ) + with pytest.warns(UserWarning, match="affects only the pretrends"): + ImputationDiD(pretrends=True, df_convention="cluster").fit( + rep_data, survey_design=design, **base + ) def test_validation_and_transactional_set_params(self): with pytest.raises(ValueError, match="df_convention"): diff --git a/tests/test_methodology_imputation.py b/tests/test_methodology_imputation.py index 6193281a..15aa3246 100644 --- a/tests/test_methodology_imputation.py +++ b/tests/test_methodology_imputation.py @@ -451,7 +451,7 @@ def counting(self, *a, **k): with warnings.catch_warnings(): warnings.simplefilter("ignore") res = ImputationDiD(n_bootstrap=50, seed=99).fit( - panel, **base, covariates=["x1", "x2"], aggregate="all" + panel, **base, covariates=["x1", "x2"] ) finally: _Cls._build_untreated_projection = orig # type: ignore[method-assign] @@ -852,38 +852,38 @@ def _assert_inference_consistent(effects): assert n_finite > 0 def test_loo_aggregate_all_analytical(self): - """aggregate="all": LOO routes through the event-study AND group - aggregators (not just overall). ATT unchanged vs non-LOO, overall LOO SE - >= non-LOO, and both aggregation surfaces have consistent finite inference.""" + """LOO routes through the post-fit event-study AND group aggregators + (not just overall). ATT unchanged vs non-LOO, overall LOO SE >= non-LOO, + and both aggregation surfaces have consistent finite inference.""" rng = np.random.default_rng(_BASE_SEED_EQ8 + 19) panel = _make_staggered_panel( rng, cohorts=[3, 4], n_per_cohort=40, n_periods=6, tau_constant=1.0 ) - r0 = ImputationDiD(leave_one_out=False).fit(panel, aggregate="all", **self._COMMON) - r1 = ImputationDiD(leave_one_out=True).fit(panel, aggregate="all", **self._COMMON) + r0 = ImputationDiD(leave_one_out=False).fit(panel, **self._COMMON) + r1 = ImputationDiD(leave_one_out=True).fit(panel, **self._COMMON) np.testing.assert_allclose(r1.overall_att, r0.overall_att, rtol=0, atol=1e-12) assert r1.overall_se > r0.overall_se - self._assert_inference_consistent(r1.event_study_effects) - self._assert_inference_consistent(r1.group_effects) + self._assert_inference_consistent(es_dict(r1)) + self._assert_inference_consistent(group_dict(r1)) def test_loo_aggregate_all_bootstrap(self): - """aggregate="all" under the multiplier bootstrap: event-study and group - inference fields are populated and consistent.""" + """Under the multiplier bootstrap the event-study and group surfaces + are materialized at fit: populated, consistent, and relayed post-fit.""" rng = np.random.default_rng(_BASE_SEED_EQ8 + 20) panel = _make_staggered_panel(rng, cohorts=[3, 4], n_per_cohort=30, n_periods=6) with warnings.catch_warnings(): warnings.simplefilter("ignore") - r = ImputationDiD(leave_one_out=True, n_bootstrap=99, seed=3).fit( - panel, aggregate="all", **self._COMMON - ) + r = ImputationDiD(leave_one_out=True, n_bootstrap=99, seed=3).fit(panel, **self._COMMON) assert np.isfinite(r.overall_se) self._assert_inference_consistent(r.event_study_effects) self._assert_inference_consistent(r.group_effects) + self._assert_inference_consistent(es_dict(r)) + self._assert_inference_consistent(group_dict(r)) def test_loo_aggregate_all_analytical_survey(self): - """aggregate="all" with an analytical survey design (weights + PSU): the - LOO rescale composes through the survey TSL variance with consistent - inference on every surface.""" + """Analytical survey design (weights + PSU): the LOO rescale composes + through the survey TSL variance with consistent inference on every + post-fit surface.""" from diff_diff.survey import SurveyDesign rng = np.random.default_rng(_BASE_SEED_EQ8 + 21) diff --git a/tests/test_naming_guard.py b/tests/test_naming_guard.py index d466e012..5df6ab08 100644 --- a/tests/test_naming_guard.py +++ b/tests/test_naming_guard.py @@ -1263,7 +1263,8 @@ def _token_family_code_refs(tok): # migrate-first rule; stale sites were migrated in the same diff that # added these entries - the CallawaySantAnna / EfficientDiD / StackedDiD / # dCDH / HAD teaching with the 4.0 PR 3b removal of their fit-time - # keyword). Dated phase-closure records in the paper-review doc describe + # keyword, the ImputationDiD / TwoStageDiD / ContinuousDiD teaching with + # PR 3c). Dated phase-closure records in the paper-review doc describe # the API as-shipped at closure time (the released-record convention) and # are not regenerated for a deprecation sweep. ("aggregate", "diff_diff/guides/llms-autonomous.txt"): ( @@ -1273,21 +1274,22 @@ def _token_family_code_refs(tok): ), ("aggregate", "diff_diff/guides/llms.txt"): ( "canonical post-fit results.aggregate() vocabulary; the fit-time " - "aggregate= teaching for CS/EfficientDiD/StackedDiD/dCDH/HAD was " - "removed with the 4.0 PR 3b sweep (M-020/M-023/M-024/M-026/M-027, " - "M-117/M-120, M-139); the ImputationDiD/TwoStageDiD/ContinuousDiD " - "clauses describe their still-deprecated keyword until PR 3c, and " - "TripleDifference's canonical staggered-mode keyword survives " - "(M-140/M-141)" + "aggregate= teaching for the eight adopters was removed with the " + "4.0 PR 3b/3c sweep (M-020..M-027, M-117..M-120, M-139); the only " + "surviving fit-time aggregate= is TripleDifference's canonical " + "staggered-mode keyword (M-140/M-141)" ), ("aggregate", "diff_diff/guides/llms-full.txt"): ( "canonical post-fit results.aggregate() vocabulary and the " - "CS/EfficientDiD/StackedDiD/dCDH/HAD signature blocks WITHOUT the " - "removed fit-time lines (4.0 PR 3b sweep; pinned by " - "tests/test_guides.py's *_fit_block_has_no_fit_time_aggregate " - "tests); the ImputationDiD/TwoStageDiD/ContinuousDiD blocks still " - "carry their DEPRECATED shim lines until PR 3c, and TripleDifference's " - "fit-time aggregate= survives (M-140/M-141)" + "signature blocks WITHOUT the removed fit-time lines (4.0 PR 3b/3c " + "sweep; pinned by tests/test_guides.py's " + "*_fit_block_has_no_fit_time_aggregate tests); the only surviving " + "fit-time aggregate= is TripleDifference's (M-140/M-141)" + ), + ("aggregate", "diff_diff/guides/llms-practitioner.txt"): ( + "canonical post-fit results.aggregate() vocabulary; the bootstrapped " + "ImputationDiD/TwoStageDiD fit-time route was replaced by the " + "4.0 PR 3c relay wording" ), ("aggregate", "docs/methodology/REGISTRY.md"): ( "the aggregate-postfit register Notes (M-020..M-027) teach the " @@ -1303,8 +1305,8 @@ def _token_family_code_refs(tok): ), ("aggregate", "docs/methodology/REPORTING.md"): ( "canonical post-fit aggregate() mention only (zero fit-time kwarg " - "sites; the CS deprecated-route clause was removed with the 4.0 PR 3b " - "sweep and the fit-time-population clause was reworded with M-027)" + "sites since the 4.0 PR 3b/3c sweep removed the deprecated-route " + "clauses; the fit-time-population clause was reworded with M-027)" ), ("aggregate", "docs/methodology/papers/dechaisemartin-2026-review.md"): ( "dated phase-closure records describing the API as-shipped at " @@ -1318,7 +1320,8 @@ def _token_family_code_refs(tok): "with M-027)" ), ("aggregate", "docs/methodology/variance-conventions.md"): ( - "canonical post-fit aggregate() vocabulary only (zero fit-time " "kwarg sites)" + "canonical post-fit aggregate() vocabulary only (zero fit-time kwarg " + "sites since the 4.0 PR 3c sweep removed the deprecated-route clause)" ), **{ ( diff --git a/tests/test_practitioner.py b/tests/test_practitioner.py index b67861a9..53805937 100644 --- a/tests/test_practitioner.py +++ b/tests/test_practitioner.py @@ -644,7 +644,11 @@ def test_aggregation_step_bootstrap_branch(self): steps = self._agg_step(output) assert len(steps) == 1 assert "BOOTSTRAPPED" in steps[0]["why"] - assert "aggregate='all'" in steps[0]["code"] + # 4.0 (PR 3c): bootstrapped fits RELAY the surfaces computed at fit; the + # deprecated fit-time aggregate='all' route is gone from the guidance. + assert "relay" in steps[0]["why"] + assert "aggregate='all'" not in steps[0]["code"] + assert "results.aggregate('event_study')" in steps[0]["code"] def test_aggregation_step_name_is_non_steps_key(self): from diff_diff.practitioner import STEPS @@ -683,9 +687,8 @@ class TestContinuousAggregationStep: ``aggregate('group')`` (not in ContinuousDiD's supported set), simulates bootstrap via a ``bootstrap_results`` field (ContinuousDiDResults keys on ``n_bootstrap`` and has no such field), - and expects an ``aggregate='all'`` fallback (never a valid - ContinuousDiD value). ContinuousDiD's step is a single unconditional - step whose wording carries the bootstrap carve-out. + and never had an ``aggregate='all'`` fallback. ContinuousDiD's step is a + single unconditional step whose wording names the bootstrapped-fit relay. """ @staticmethod @@ -706,8 +709,9 @@ def test_aggregation_step_present_with_all_routes(self): assert "results.aggregate('event_study')" in steps[0]["code"] assert "results.aggregate('dose')" in steps[0]["code"] assert "results.aggregate('simple')" in steps[0]["code"] - # The bootstrap carve-out is carried in the wording. - assert "n_bootstrap=0" in steps[0]["why"] + # 4.0 (PR 3c): bootstrapped fits relay the surface computed at fit. + assert "relays" in steps[0]["why"] + assert "n_bootstrap=0" not in steps[0]["why"] def test_aggregation_step_name_is_non_steps_key(self): from diff_diff.practitioner import STEPS diff --git a/tests/test_pretrends_event_study.py b/tests/test_pretrends_event_study.py index 0ee5b3ea..a5c7acd6 100644 --- a/tests/test_pretrends_event_study.py +++ b/tests/test_pretrends_event_study.py @@ -7,27 +7,11 @@ import numpy as np import pandas as pd -import pytest from legacy_surface import es_dict from diff_diff.imputation import ImputationDiD from diff_diff.two_stage import TwoStageDiD -# --------------------------------------------------------------------------- -# Rows M-021/M-022 (+ M-118/M-119): ImputationDiD / TwoStageDiD -# ``fit(aggregate=, balance_e=)`` is deprecated (3.9, removed at the 4.0 cut) and -# warns on ANY supplied value. Analytic sites read the post-fit -# ``results.aggregate(...)`` container; the BOOTSTRAPPED sites below stay on the -# fit-time route until the 4.0 relay lands (the post-fit route raises -# NotImplementedError on a bootstrapped Imputation/TwoStage fit today). The shim -# warning is therefore filtered BY MESSAGE, scoped to -# these two estimators only - every other FutureWarning (including the other -# estimators' aggregate() shims) still surfaces. -# --------------------------------------------------------------------------- -pytestmark = pytest.mark.filterwarnings( - r"ignore:(ImputationDiD|TwoStageDiD)\.fit\((aggregate=|balance_e=|aggregate= / balance_e=)\):FutureWarning" -) - def generate_test_data( n_units: int = 100, @@ -546,12 +530,9 @@ def test_imputation_pretrends_with_covariates(self): time="time", first_treat="first_treat", covariates=["x1"], - aggregate="event_study", ) - negative = { - h: v for h, v in results.event_study_effects.items() if h < -1 and v["n_obs"] > 0 - } + negative = {h: v for h, v in es_dict(results).items() if h < -1 and v["n_obs"] > 0} assert len(negative) > 0, "Should have pre-period horizons" for h, eff in negative.items(): assert np.isfinite(eff["se"]), f"h={h}: SE not finite with covariates" @@ -572,7 +553,6 @@ def test_imputation_pretrends_bootstrap_post_only(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) # With bootstrap @@ -582,15 +562,16 @@ def test_imputation_pretrends_bootstrap_post_only(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) - # Pre-period effects and SEs should be identical (bootstrap doesn't touch them) - for h in results_no_boot.event_study_effects: - if h >= 0 or results_no_boot.event_study_effects[h]["n_obs"] == 0: + # Pre-period effects and SEs should be identical (bootstrap doesn't + # touch them): the analytical recompute vs the relayed fit-time surface. + es_no_boot, es_boot = es_dict(results_no_boot), es_dict(results_boot) + for h in es_no_boot: + if h >= 0 or es_no_boot[h]["n_obs"] == 0: continue - eff_nb = results_no_boot.event_study_effects[h] - eff_b = results_boot.event_study_effects[h] + eff_nb = es_no_boot[h] + eff_b = es_boot[h] np.testing.assert_allclose( eff_nb["effect"], eff_b["effect"], @@ -692,7 +673,6 @@ def test_nondefault_index_bootstrap(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) rng = np.random.default_rng(42) @@ -704,14 +684,14 @@ def test_nondefault_index_bootstrap(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) - for h in results_default.event_study_effects: - if results_default.event_study_effects[h]["n_obs"] == 0: + es_default, es_perm = es_dict(results_default), es_dict(results_perm) + for h in es_default: + if es_default[h]["n_obs"] == 0: continue - eff_d = results_default.event_study_effects[h] - eff_p = results_perm.event_study_effects[h] + eff_d = es_default[h] + eff_p = es_perm[h] np.testing.assert_allclose( eff_d["effect"], eff_p["effect"], @@ -779,12 +759,9 @@ def test_two_stage_pretrends_bootstrap(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) - negative = { - h: v for h, v in results.event_study_effects.items() if h < -1 and v["n_obs"] > 0 - } + negative = {h: v for h, v in es_dict(results).items() if h < -1 and v["n_obs"] > 0} assert len(negative) > 0, "Should have pre-period horizons" for h, eff in negative.items(): assert np.isfinite(eff["se"]), f"h={h}: SE not finite" diff --git a/tests/test_survey_phase8.py b/tests/test_survey_phase8.py index 46923a30..54164992 100644 --- a/tests/test_survey_phase8.py +++ b/tests/test_survey_phase8.py @@ -11,21 +11,6 @@ compute_replicate_vcov, ) -# --------------------------------------------------------------------------- -# Rows M-021/M-022 (+ M-118/M-119): ImputationDiD / TwoStageDiD -# ``fit(aggregate=, balance_e=)`` is deprecated (3.9, removed 4.0) and warns on -# ANY supplied value. The deprecated fit-time route is kept DELIBERATELY here: -# these tests pin FIT-TIME surface behaviour (bit-equality grids, bootstrap -# aggregation, R/Stata parity, replicate overrides, native effect dicts) that -# the post-fit ``results.aggregate(...)`` container route does not reproduce -# shape-for-shape. The shim warning is therefore filtered BY MESSAGE, scoped to -# these two estimators only - every other FutureWarning (including the other -# estimators' aggregate() shims) still surfaces. -# --------------------------------------------------------------------------- -pytestmark = pytest.mark.filterwarnings( - r"ignore:(ImputationDiD|TwoStageDiD)\.fit\((aggregate=|balance_e=|aggregate= / balance_e=)\):FutureWarning" -) - # --------------------------------------------------------------------------- # Fixtures # --------------------------------------------------------------------------- @@ -1311,11 +1296,11 @@ def test_pretrends_survey_with_covariates(self, staggered_data): first_treat="first_treat", covariates=["x1"], survey_design=sd, - aggregate="event_study", ) - pre = [h for h in result.event_study_effects if h < -1] - for h in pre: - assert np.isfinite(result.event_study_effects[h]["se"]) + es = result.aggregate("event_study") + pre = es.event_time < -1 + assert pre.any() + assert np.all(np.isfinite(es.se[pre])) def test_pretrends_survey_with_bootstrap(self, staggered_data): """Survey pretrends + bootstrap: bootstrap doesn't overwrite pre-period SEs.""" @@ -1331,12 +1316,19 @@ def test_pretrends_survey_with_bootstrap(self, staggered_data): time="time", first_treat="first_treat", survey_design=sd, - aggregate="event_study", ) + # Bootstrapped fits materialize the surface at fit time. + assert result.event_study_effects is not None pre = [h for h in result.event_study_effects if h < -1] + assert pre for h in pre: # Pre-period SEs come from lead regression, not bootstrap assert np.isfinite(result.event_study_effects[h]["se"]) + # The relay reproduces the stored surface. + es = result.aggregate("event_study") + for h in pre: + i = list(es.event_time).index(h) + assert es.se[i] == result.event_study_effects[h]["se"] def test_pretrends_survey_always_treated_psu(self): """Survey pretrends with a PSU/stratum that has no untreated obs.""" diff --git a/tests/test_two_stage.py b/tests/test_two_stage.py index 68fbd356..e236ab96 100644 --- a/tests/test_two_stage.py +++ b/tests/test_two_stage.py @@ -15,21 +15,6 @@ TwoStageDiDResults, ) -# --------------------------------------------------------------------------- -# Rows M-021/M-022 (+ M-118/M-119): ImputationDiD / TwoStageDiD -# ``fit(aggregate=, balance_e=)`` is deprecated (3.9, removed 4.0) and warns on -# ANY supplied value. The deprecated fit-time route is kept DELIBERATELY here: -# these tests pin FIT-TIME surface behaviour (bit-equality grids, bootstrap -# aggregation, R/Stata parity, replicate overrides, native effect dicts) that -# the post-fit ``results.aggregate(...)`` container route does not reproduce -# shape-for-shape. The shim warning is therefore filtered BY MESSAGE, scoped to -# these two estimators only - every other FutureWarning (including the other -# estimators' aggregate() shims) still surfaces. -# --------------------------------------------------------------------------- -pytestmark = pytest.mark.filterwarnings( - r"ignore:(ImputationDiD|TwoStageDiD)\.fit\((aggregate=|balance_e=|aggregate= / balance_e=)\):FutureWarning" -) - # ============================================================================= # Shared test data generation # ============================================================================= @@ -310,13 +295,14 @@ def test_to_dataframe_invalid_level(self): results.to_dataframe("invalid") def test_to_dataframe_no_event_study(self): - """Test to_dataframe raises when event study not computed.""" + """to_dataframe raises on an analytic fit (no fit-time event-study + surface) and steers to the post-fit aggregate() route.""" data = generate_test_data() results = TwoStageDiD().fit( data, outcome="outcome", unit="unit", time="time", first_treat="first_treat" ) - with pytest.raises(ValueError, match="Event study effects not computed"): + with pytest.raises(ValueError, match="not stored on this fit"): results.to_dataframe("event_study") def test_repr(self): @@ -1306,13 +1292,19 @@ def test_bootstrap_event_study(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) assert results.bootstrap_results is not None assert results.bootstrap_results.event_study_ses is not None for h, se in results.bootstrap_results.event_study_ses.items(): assert se > 0 + # Bootstrapped fits materialize the percentile surface at fit time and + # aggregate() relays it. + assert results.event_study_effects is not None + es = results.aggregate("event_study") + for h, se in zip(es.event_time, es.se): + if h in results.bootstrap_results.event_study_ses: + assert se == results.event_study_effects[h]["se"] def test_bootstrap_weights_mammen(self, ci_params): """Bootstrap with mammen weights should produce valid results.""" @@ -1352,7 +1344,6 @@ def test_bootstrap_weights_event_study(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) br = results.bootstrap_results @@ -1373,7 +1364,6 @@ def test_bootstrap_weights_group(self, ci_params): unit="unit", time="time", first_treat="first_treat", - aggregate="group", ) br = results.bootstrap_results @@ -1790,9 +1780,10 @@ def test_bootstrap_dropped_coefficient_propagates_nan_inference(self): unit="unit", time="time", first_treat="first_treat", - aggregate="all", ) + # Bootstrapped fits materialize the percentile surface at fit time. es = res.event_study_effects + assert es is not None nan_h = [h for h, e in es.items() if np.isnan(e["se"])] fin_h = [h for h, e in es.items() if np.isfinite(e["se"]) and e["se"] > 0] assert nan_h, ( @@ -2216,23 +2207,45 @@ def _fit_ts(est, data, **kw): def _assert_results_bit_equal(r0, r1): - """Assert two TwoStageDiDResults are numerically identical (NaN-aware).""" + """Assert two TwoStageDiDResults are numerically identical (NaN-aware). + + Bootstrapped fits materialize the percentile ``event_study_effects`` / + ``group_effects`` at fit time, so those legacy fields are compared + directly; analytic fits store no fit-time surface, so the post-fit + ``aggregate()`` containers are compared instead. + """ def _eq(a, b): return a == b or (np.isnan(a) and np.isnan(b)) assert _eq(r0.overall_att, r1.overall_att) assert _eq(r0.overall_se, r1.overall_se) + assert (r0.bootstrap_results is None) == (r1.bootstrap_results is None) + if r0.bootstrap_results is not None: + for attr in ("event_study_effects", "group_effects"): + e0 = getattr(r0, attr) + e1 = getattr(r1, attr) + assert e0 is not None and e1 is not None, f"bootstrapped fit left {attr} None" + assert set(e0) == set(e1) + for k in e0: + for f in ("effect", "se"): + assert _eq(e0[k][f], e1[k][f]), f"{attr}[{k}][{f}] differs" + return for attr in ("event_study_effects", "group_effects"): - e0 = getattr(r0, attr) - e1 = getattr(r1, attr) - if e0 is None: - assert e1 is None - continue - assert set(e0) == set(e1) - for k in e0: - for f in ("effect", "se"): - assert _eq(e0[k][f], e1[k][f]), f"{attr}[{k}][{f}] differs" + assert getattr(r0, attr) is None and getattr(r1, attr) is None + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + pairs = [ + (level, r0.aggregate(level), r1.aggregate(level)) for level in ("event_study", "group") + ] + for level, c0, c1 in pairs: + labels = "event_time" if level == "event_study" else "label" + np.testing.assert_array_equal(getattr(c0, labels), getattr(c1, labels)) + for f in ("att", "se"): + # Exact (NaN-aware) equality: vcov_type="hc1" is a pure no-op. + np.testing.assert_array_equal( + getattr(c0, f), getattr(c1, f), err_msg=f"{level}.{f} differs" + ) def _assert_full_bootstrap_nan(r): @@ -2375,41 +2388,37 @@ def test_cluster_name_suppressed_under_replicate_survey(self): # ---- bit-equality regression guards (vcov_type='hc1' is a pure no-op) ---- - @pytest.mark.parametrize("aggregate", [None, "event_study", "group", "all"]) - def test_default_path_bit_equal(self, aggregate): + def test_default_path_bit_equal(self): data = generate_test_data(n_units=80, seed=10) - r0 = _fit_ts(TwoStageDiD(), data, aggregate=aggregate) - r1 = _fit_ts(TwoStageDiD(vcov_type="hc1"), data, aggregate=aggregate) + r0 = _fit_ts(TwoStageDiD(), data) + r1 = _fit_ts(TwoStageDiD(vcov_type="hc1"), data) _assert_results_bit_equal(r0, r1) - @pytest.mark.parametrize("aggregate", [None, "event_study", "group", "all"]) - def test_cluster_path_bit_equal(self, aggregate): + def test_cluster_path_bit_equal(self): data = generate_test_data(n_units=80, seed=11) data["st"] = data["unit"] % 7 - r0 = _fit_ts(TwoStageDiD(cluster="st"), data, aggregate=aggregate) - r1 = _fit_ts(TwoStageDiD(cluster="st", vcov_type="hc1"), data, aggregate=aggregate) + r0 = _fit_ts(TwoStageDiD(cluster="st"), data) + r1 = _fit_ts(TwoStageDiD(cluster="st", vcov_type="hc1"), data) _assert_results_bit_equal(r0, r1) def test_tsl_survey_path_bit_equal(self): data, _ = _add_survey_cols(generate_test_data(n_units=80, seed=12)) design = SurveyDesign(weights="w") - r0 = _fit_ts(TwoStageDiD(), data, survey_design=design, aggregate="all") - r1 = _fit_ts(TwoStageDiD(vcov_type="hc1"), data, survey_design=design, aggregate="all") + r0 = _fit_ts(TwoStageDiD(), data, survey_design=design) + r1 = _fit_ts(TwoStageDiD(vcov_type="hc1"), data, survey_design=design) _assert_results_bit_equal(r0, r1) def test_replicate_survey_path_bit_equal(self): data, rep_cols = _add_survey_cols(generate_test_data(n_units=80, seed=13)) design = SurveyDesign(weights="w", replicate_weights=rep_cols, replicate_method="JK1") - r0 = _fit_ts(TwoStageDiD(), data, survey_design=design, aggregate="event_study") - r1 = _fit_ts( - TwoStageDiD(vcov_type="hc1"), data, survey_design=design, aggregate="event_study" - ) + r0 = _fit_ts(TwoStageDiD(), data, survey_design=design) + r1 = _fit_ts(TwoStageDiD(vcov_type="hc1"), data, survey_design=design) _assert_results_bit_equal(r0, r1) def test_bootstrap_path_bit_equal(self): data = generate_test_data(n_units=80, seed=14) - r0 = _fit_ts(TwoStageDiD(n_bootstrap=199, seed=99), data, aggregate="all") - r1 = _fit_ts(TwoStageDiD(n_bootstrap=199, seed=99, vcov_type="hc1"), data, aggregate="all") + r0 = _fit_ts(TwoStageDiD(n_bootstrap=199, seed=99), data) + r1 = _fit_ts(TwoStageDiD(n_bootstrap=199, seed=99, vcov_type="hc1"), data) _assert_results_bit_equal(r0, r1) b0, b1 = r0.bootstrap_results, r1.bootstrap_results assert b0.overall_att_se == b1.overall_att_se @@ -2447,12 +2456,7 @@ def test_bootstrap_single_cluster_returns_nan(self): unit="unit", time="time", first_treat="first_treat", - # Fit-time aggregation kept on this BOOTSTRAPPED fit until the - # 4.0 relay: the NaN guard must cover the per-horizon and - # per-group payloads, not just the overall quintet. - aggregate="all", ) - assert r.event_study_effects and r.group_effects _assert_full_bootstrap_nan(r) def test_bootstrap_single_psu_survey_returns_nan(self): @@ -2468,9 +2472,7 @@ def test_bootstrap_single_psu_survey_returns_nan(self): time="time", first_treat="first_treat", survey_design=design, - aggregate="event_study", # bootstrapped: fit-time route until the 4.0 relay ) - assert r.event_study_effects _assert_full_bootstrap_nan(r) # ---- metadata reflects the post-drop fit sample (codex P2) ---- @@ -2648,16 +2650,22 @@ def test_analytical_aggregate_vcov_diag_matches_ses(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) - assert res.event_study_vcov is not None - assert res.event_study_vcov_index is not None - ses = {h: d["se"] for h, d in res.event_study_effects.items()} - diag = np.sqrt(np.maximum(np.diag(res.event_study_vcov), 0.0)) - for i, h in enumerate(res.event_study_vcov_index): + es = res.aggregate("event_study") + # Analytic fits store no fit-time surface; the GMM matrix travels on + # the container. + assert res.event_study_vcov is None + assert res.event_study_vcov_index is None + assert es.vcov is not None + assert es.vcov_index is not None + ses = dict(zip(es.event_time.tolist(), es.se.tolist())) + diag = np.sqrt(np.maximum(np.diag(es.vcov), 0.0)) + for i, h in enumerate(es.vcov_index.tolist()): np.testing.assert_allclose(diag[i], ses[h], rtol=1e-14) # Non-survey GMM inference is normal-theory: no df provenance. assert res.event_study_df is None + assert es.df_survey is None + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) def test_bootstrap_clears_vcov_and_df(self): # Bootstrap replaces the stored ES se/p/CI with percentile values: @@ -2671,12 +2679,18 @@ def test_bootstrap_clears_vcov_and_df(self): unit="unit", time="time", first_treat="first_treat", - aggregate="event_study", ) + es = res.aggregate("event_study") assert res.bootstrap_results is not None + assert res.event_study_effects is not None assert res.event_study_vcov is None assert res.event_study_vcov_index is None assert res.event_study_df is None + # The relay carries neither matrix nor df. + assert es.vcov is None + assert es.vcov_index is None + assert es.df_survey is None + assert np.all(np.isnan(np.asarray(es.df, dtype=float))) def test_replicate_weight_survey_clears_vcov_and_threads_final_df(self): # Replicate-weight designs: reported ES SEs come from the replicate