[anth-volk]

Fixes #682

Base: main. #615 merged into main on 2026-10-20 (2591af3e), so main was merged into this branch (36c3d8d3, a merge, not a rebase) and the base changed from uk-553-area-routing-tooling to main. Do not merge before both of these:

1. Microcosm's release writer change (H2c) defines the new release manifest and confirms the shape assumed below;
2. Microcosm's one-time migration (P1) has moved the existing UK builds into their own folders and written releases.json.

Until a manifest in the new layout exists, nothing here changes behaviour: every manifest published today has build as an object and is read exactly as before.

Why

Microcosm is moving to a layout where:

• each build is published once in its own folder, releases/<release>/<build>/, and never changed;
• a root releases.json (microcosm-emitter#24, microcosm_emitter_interface.releases) records every published build of each release, which one is current, and which are withdrawn;
• no new Hugging Face tags are made.

#615 names a build by build.cut_tag, which new builds don't have, and it assumes build is an object. A new manifest names its release and build as strings, so without this PR it fails to parse.

What changes

Reading the manifest (provenance/manifest.py)

• DataReleaseManifest.release is new. A manifest with string release and build is read with build as DataBuildInfo(build_id=<build>). Both must match the publisher's identifier rule, ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$. Dumps write build back as the string, so merges round-trip it.
• release_build_id is unchanged: such a build has no cut_tag, so it returns the build name, and two builds of one release never match.
• A string build without release, or release with an object build, is refused.
• build_folder returns releases/<release>/<build>, or None for older manifests.
• DataReleaseArtifact.revision may be absent only in a build-folder manifest. Older manifests still need it on every artifact.

Certifying (provenance/certification.py, through the shared read_build_folder)

• The manifest URI must be the build's own folder, releases/<release>/<build>/release_manifest.json; a manifest read from any other path is refused.
• The repository's releases.json at main must list the build and must not mark it withdrawn, or certification refuses it. A local-area release passed with --regional-manifest-uri is checked the same way. The record is read at main, not at the URI's revision, because a build is withdrawn after the commit that published it. Which build is current doesn't matter: a certificate pins one build.
• The whole record must also be one the release writer's parser accepts (ReleasesRecord.parse with check_record, microcosm-emitter#24, matched at 35481d4d). .py repeats those rules without depending on the emitter interface package: exact keys at every level, schema_version 1, safe release and build names, valid roles, a non-empty published_at and a boolean withdrawn on every build, a default that is null or a listed release, a current that is null or a listed build that isn't withdrawn, and at most 4 MiB. So .py never certifies from a record the writer and its other readers refuse.
• The build's own files (its artifacts in the manifest's repository) are pinned at releases/<release>/<build>/<file> and at the URI's revision.
  • Each one's path must be a bare file name, alone or after the folder, which is what the release writer publishes. A nested path, a . or .. segment, or a backslash is refused. HTTP clients remove dot segments, so <folder>/../<other build>/<file> would otherwise fetch another build's file.
  • Why the URI's revision: the publisher writes a build's files and its manifest in one commit and never changes them, so any revision that holds the manifest holds the same files. An artifact's own revision field can't name that commit.
  • The docs recommend certifying at the commit that published the build. main resolves to the same files, because the folder never changes.
• Artifacts from other repositories keep their own paths and pins.
• The pinned paths flow into its datasets, the default national region template, a local-area release routed with --regional-manifest-uri, its certified dataset URI, and the reachability checks.
• The build's name becomes data_releases.uk.version, the certificate's data_build_id and the changelog fragment name. These record the build name, not <release>/<build>: the bundle already pins the release through release_manifest_path, and a slash in the name would put the fragment in a subfolder.

Import-time check

• For a build in its own folder, get_data_release_manifest requires the remote manifest to name the release and build of the bundle's pinned release_manifest_path. It then pins the build's files the same way certification does, with the same file-name rule.
• A mismatch raises the new ReleaseBuildFolderError, a ValueError. It is deliberately not a DataReleaseManifestUnavailableError, so the offline fallback to the bundled certificate never covers it.
• Import doesn't read releases.json. A build withdrawn after it was certified keeps loading in every installed .py release whose bundle pins it; certifying again changes only the .py releases published afterwards. Whether an installed bundle should refuse it is not decided here.

Docs and changelog: docs/engineering/skills/data-certification.md has a new section, "Builds in their own release folders". The fragment is changelog.d/682.added.md.

Assumed until H2c defines the manifest

Item	Assumption
Identity	Top-level release and build are strings equal to the folder's names. emitter#24's check_repository enforces exactly this. Readers tell the layouts apart by whether build is a string.
Files	Each file is an artifacts entry with kind, path, repo_id and sha256. path is a bare file name, as the emitter writes it, or that name after releases/<release>/<build>/. revision may be dropped. A role (for example dataset or calibration_diagnostics) may be present; this PR doesn't use it.
Unchanged fields	default_datasets, dataset_role, is_default and compatible_model_packages / compatible_core_packages keep their current meaning. Manifest-declared metadata.region_datasets templates are copied as they are, not repinned; Microcosm writes none today.

Open for H2c: today's build object also carries built_at, built_with_model_package (version, git SHA, data build fingerprint) and built_with_core_package. With build a string, these have no place yet, so a build-folder manifest records none of them. As a result:

• the built_with_model_package and matching_data_build_fingerprint bases can't apply;
• certification relies on compatible_model_packages / compatible_core_packages or a qualified pair;
• the TRO's start time falls back to nothing.

H2c needs to say where these details go. This PR doesn't invent a place for them.

Tests

All new tests are in tests/test_release_build_folders.py and use fixtures only, with no network. Each was written first and run before the change that makes it pass.

Commit	Red (tests first)	Green
9a5286c2 Read a build published in its own release folder	12 new tests: 10 failed, 2 passed (both guard behaviour that already held)	12 passed; re-run 12 passed
158b5aee Certify a build from its own release folder	10 new tests: 6 failed, 4 passed (the recorded manifest path, the build's name and other-repository pins already held after the first commit, plus one tagged-release regression test)	22 passed; re-run 22 passed
0320e5cc Refuse at import a manifest that names another build than its folder	5 new tests: 3 failed, 2 passed (the build already named the certificate after the first commit)	27 passed; re-run 27 passed
36c3d8d3 Merge main (with #615's final code)	Not applicable: a merge with no conflicts	27 passed
edc5356b red / ea01e115 green: certify a build only while releases.json lists it and has not withdrawn it	6 new tests: 5 failed (four DID NOT RAISE CertificationError, one because releases.json was never requested), 1 passed (tagged releases read no record); 33 in the file	33 passed
ef297cb6 red / 62419805 green: accept only bare file names directly inside a build's folder	7 new tests: 7 failed (DID NOT RAISE); 40 in the file	40 passed
6b72ec01 Document the releases.json and file-name checks	Not applicable: documentation only	40 passed
58d43c6f red / 90301909 green: certify only from a releases.json record the release writer accepts	28 new tests: 22 failed (all DID NOT RAISE CertificationError), 6 passed (3 records already refused: missing schema_version, schema_version 2, missing releases; 3 records the writer accepts and certification must keep accepting); 68 in the file. The record fixture now keeps current on a listed, non-withdrawn build, so the existing withdrawn and unlisted tests use records the writer accepts.	68 passed
90301909 also corrects the guide's sentence on withdrawn builds at import	Not applicable: a documentation correction needs no red step	68 passed

Following the agreed local-testing rule, only the new test file was run locally. #615's test files (test_certify_data_release.py, test_release_manifests.py, test_qualified_pair_certification.py) are left to CI. They use manifests whose build is an object, which never read releases.json. ruff format --check and ruff check pass on the changed files. mypy on the two changed modules reports only errors that predate this PR: missing requests stubs, and three no-any-return errors in functions this PR doesn't touch.

The releases.json records in the new tests were also run, outside this repository, through the writer's own ReleasesRecord.parse at microcosm-emitter#24's 35481d4d. It refuses every record the tests expect certification to refuse and accepts the others, and .py's check gives the same result for all 30. The writer also accepts every record the fixture builds for the existing withdrawn and unlisted tests.

Review round 2026-10-22

• Compared with microcosm-emitter#24 at 7a47f9a (its head after a rebase; 35481d4d above no longer exists). The releases.json rules in _read_releases_record match the writer's ReleasesRecord.parse and check_record exactly: the exact keys at every level, the integer schema_version 1, the release and build name pattern, the role pattern, published_at and withdrawn types, current and default, the 4 MiB limit, and unreadable or too deeply nested JSON. The writer's rules have not changed since 20 October. The file-name rule matches bare_file_name, and the manifest identity check matches check_repository.
• No bugs found.
• 89c2cff adds tests for the rules most likely to drift: schema_version true or 1.0, and a record that is truncated JSON, not text, or nested too deeply. They pass without a code change; the fake response now decodes text the way requests does. 73 tests in the file pass.
• 72347a4 corrects the ReleaseBuildFolderError docstring, which now also covers releases.json refusals.
• ruff format --check and ruff check pass; mypy on the two changed modules reports only the two no-any-return errors that predate this PR.

🤖 Generated with Claude Code