[MaxGhenis]

Require audited evidence for a UK regional release served on an accepted model mismatch.

This branch stacks #704 (da2ae8fb) into #699 (afee349d2) with a merge commit. The intended release order remains main → #704 → #699 → certification.

Invariants:

• National certification never widens. The national dataset name/hash, model coverage and core checks remain separate from the regional exception.
• An accepted area mismatch requires a reviewer reason and audited evidence bound to the exact dense cut URI, original manifest bytes, artifact identity/SHA-256, and exact target model package/version. Certification recomputes the comparison and refuses every dropped column. There is no dropped-column override.
• Every defaulted input is recorded across every stored period, accounting for the country loader's input aliases. Free-text limitations and the dense cut's pre-fix VAT limitation survive Pydantic validation and are hashed with the bundle.
• TRACE and execution receipts expose the full regional identity, acceptance, reason, input audit and limitations. TRACE preserves the record as a JSON-LD JSON literal; area receipts remain uncertified and identify their actual build model.
• Re-certifying with --drop-regional-release removes the exception, regional manifest digest, area routes and dense artifact without changing the national certificate.

The comparison is ported from Microcosm #1348's tools/certify_uk_cut_in_policyengine_py.py and strengthened to inspect byte-verified H5 entity-table columns by period. scripts/audit_regional_input_surface.py produces the required JSON; certify-data --regional-input-surface-evidence consumes it with the accepted reason. The original regional manifest's raw-byte digest is retained.

Budget route: national microcosm-uk-2024-25-national-20261023T075039Z-a133f37a (2.147.2) and dense microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe (built on 2.132.6). The exact model-inventory comparison yields 16 added pure inputs, including salary_sacrifice_returned_to_income and local_bus_uncapped_single_fare; the complete list is documented. Actual dense-byte evidence must be generated by certification with authenticated access to the H5. This implementation job has no HF credential and does not certify that pair.

Required downstream refresh: update the dashboard's vendored tests/fixtures/w4/execution_receipt.v1.schema.json from this repository's regenerated schema.

Validation: 1,248 selected cases passed and two skipped across the UK run and three successful US metadata reruns. One unrelated US offline-import integration case was explicitly deselected. Formatting, lint, generated-schema equality and bundle checks pass. Mypy retains the same 24 baseline diagnostics, with zero errors in the new modules. Expanded validation was interrupted after 3,008 reported passes and five skips; it is not claimed complete. Details are recorded in .job/HANDOFF.md. Regression tests demonstrated failures before implementation. Leave this PR draft.

🤖 Generated with Claude Code

[juaristi22]

My ruling on the accepted mismatch: keep the evidence, and fix what counts as dropped.

I like requiring evidence for this exception, and producing it is cheap. I dry-ran this PR's own scripts/audit_regional_input_surface.py (at c669bbd9) on the 19 Oct dense cut, microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe, with policyengine-uk 2.147.2 and 2.151.0 (core 3.33.3). The download took 34 s and each audit 17–23 s. But as written, the audit refuses every Microcosm dense cut, so this route can never be used:

• Both audits refused, on 83 "dropped" columns. None of the 83 is a policyengine-uk variable in 2.132.6 either, the version the dense cut was built with. They are Microcosm's own columns: area codes, build and source ids, clone indices and SPI donor fields. They include constituency_code_oa and la_code_oa, the columns src/policyengine/countries/uk/regions.py scopes areas by. The same audit would refuse this file against its own build model.
• Columns really lost since 2.132.6: none, under either 2.147.2 or 2.151.0.
• The defaulted list over-reports the same way. It lists 139 inputs under 2.147.2 and 160 under 2.151.0, but 123 of those already default under 2.132.6. New since the build: the 16 the docs list for 2.147.2, and 37 for 2.151.0. The extra 21 on 2.151.0 include the gambling-duty inputs, liquid and solid fuel consumption and bonus sacrifice, so area breakdowns of those measures would use defaults there. That belongs in the limitations if the route is used on 2.151.0.

The port dropped the baseline that PolicyEngine/microcosm#1348's comparison keeps: "Columns neither model defines, and inputs absent under both, are not changes."

Please:

1. Count as dropped only stored columns that the regional build model defines and the target model does not. Record never-modelled columns as information. This needs the build model's variables too; Let anyone build, sign and certify a Microcosm UK cut microcosm#1348 reads them from a throwaway uv run --with policyengine-uk==<version> environment.
2. Report defaulted inputs relative to the build model: target inputs the file doesn't store (after aliases) that weren't inputs of the build model.
3. Keep refusing real drops, with no override. There are none today.
4. Fix the red test: test__given_bundle_wrapper_empty_reason__then_certifier_refuses_it expects the certifier to refuse an empty reason, but bundle.py's parser now refuses it first.

With those changes the evidence takes about a minute to produce and can be scripted, which is fine for Budget day. Until the new national and dense pair on 2.151.0 lands, we accept the mismatch only to test the machinery.