Certify Microcosm builds published side by side in their own release folders

changelog.d/682.added.md

@@ -0,0 +1 @@
+Certify Microcosm builds published side by side in their own release folders (`releases/<release>/<build>/`, with no Hugging Face tag): a release manifest that names its `release` and `build` as strings is certified from that folder, its own files are pinned inside it at the certified revision, and its `build` names the certificate, so two builds of one release never match. Certification refuses such a build unless the repository's `releases.json` at `main` lists it, has not withdrawn it, and is a record the release writer's parser accepts, and refuses any of its own files that is not a bare file name directly inside its folder. At import, a remote manifest that names another release or build than the bundle's pinned folder is refused. Manifests published before this layout are read as before.

docs/engineering/skills/data-certification.md

@@ -34,6 +34,52 @@ uses the cut tag as the build id (`data_releases.<country>.version`, the
 certification's `data_build_id` and the changelog fragment name), so two cuts
 of one line never look identical.
 
+### Builds in their own release folders
+
+Once Microcosm publishes each build in its own folder,
+`releases/<release>/<build>/`, a build is never changed after publishing and
+gets no Hugging Face tag. The repository's root `releases.json` records which
+build of each release is current. That build's `release_manifest.json` names
+its `release` and `build` as strings, for example
+`microcosm-uk-2024-25-national` and
+`uk-frs-calibration-attempt-20261019T083323Z-6cba5d71`. Certify the build from
+its own folder, at the commit that published it:
+
+```bash
+python scripts/bundle.py certify-data --country uk --data-producer populace \
+  --manifest-uri "hf://dataset/policyengine/populace-uk-private@<commit>/releases/<release>/<build>/release_manifest.json"
+```
+
+- The URI must be the build's own folder: a manifest that names another
+  release or build than the folder it was read from is refused.
+- The repository's `releases.json` at `main` must list the build and must not
+  mark it withdrawn, or certification refuses it. A local-area release passed
+  with `--regional-manifest-uri` is checked the same way. Which build is
+  `current` doesn't matter: a certificate pins one build.
+- The whole record must also be one the release writer's parser
+  (microcosm-emitter's `ReleasesRecord.parse`) accepts, or certification
+  refuses it. .py repeats the writer's rules: exactly the keys the writer
+  writes, at every level; `schema_version` 1; safe release and build names and
+  valid roles; a non-empty `published_at` and a boolean `withdrawn` on every
+  build; a `default` that is null or a listed release; a `current` that is null
+  or a listed build that isn't withdrawn; and at most 4 MiB.
+- The build's own files are pinned at `releases/<release>/<build>/<file>` and
+  at the URI's revision, whatever revision their manifest entries name. Each
+  one's `path` must be a bare file name, alone or after the folder, as the
+  release writer publishes it; a nested path, a `.` or `..` segment or a
+  backslash is refused. Artifacts from other repositories keep their own pins.
+- The certificate names the build by `build`
+  (`data_releases.<country>.version`, the certification's `data_build_id` and
+  the changelog fragment name), so two builds of one release never match.
+- At import, a remote manifest that names another release or build than the
+  bundle's pinned folder is refused, even when the bundle carries a
+  certificate. Import doesn't read `releases.json`: a build withdrawn after it
+  was certified keeps loading in every installed .py release whose bundle pins
+  it. Certifying again changes only the .py releases published afterwards.
+
+Manifests published before this layout, whose `build` is an object, are
+certified as described above.
+
 ### UK constituency and local-authority regions
 
 UK constituency and local-authority regions row-filter the dataset a regional

src/policyengine/provenance/certification.py

@@ -29,6 +29,13 @@
 Pin the cut tag (``@<cut-tag>``), never a branch: a release line's directory
 (``releases/<release-id>/``) is rewritten by each cut.
 
+A build published in its own folder (``releases/<release>/<build>/``, named
+by its manifest's ``release`` and ``build`` strings) has no tag. Certify it
+from that folder at the commit that published it; its own files are pinned
+inside the folder at that revision, and its ``build`` names the certificate.
+The repository's ``releases.json`` at ``main`` must list the build and not
+have withdrawn it, and must be a record the release writer accepts.
+
 ``scripts/bundle.py certify-data`` is the operator-facing wrapper. Network
 access is required (HF manifest fetch + PyPI wheel metadata). Countries whose
 data release predates release manifests need a dedicated data-producer strategy
@@ -54,11 +61,16 @@
 )
 from policyengine.provenance.manifest import (
     HF_REQUEST_TIMEOUT_SECONDS,
+    RELEASES_RECORD_NAME,
+    RELEASES_RECORD_REVISION,
     DataReleaseArtifact,
     DataReleaseManifest,
     RegionalReleaseKind,
+    ReleaseBuildFolderError,
     _specifier_matches,
+    check_build_listed,
     fetch_pypi_wheel_metadata,
+    read_build_folder,
     release_build_id,
 )
 
@@ -239,12 +251,53 @@ def release_manifest_dir(parts: dict) -> str:
     return parts["path"].rsplit("/", 1)[0]
 
 
+def https_releases_record_url(parts: dict) -> str:
+    """The repository's ``releases.json`` at ``main``, its current record."""
+    prefix = "datasets/" if parts["repo_type"] == "dataset" else ""
+    return (
+        f"https://huggingface.co/{prefix}{parts['repo_id']}/resolve/"
+        f"{RELEASES_RECORD_REVISION}/{RELEASES_RECORD_NAME}"
+    )
+
+
+def _require_listed_build(
+    manifest: DataReleaseManifest, parts: dict, headers: dict
+) -> None:
+    """Refuse a build in its own folder that ``releases.json`` at ``main``
+    does not list, or has withdrawn.
+
+    The record is read at ``main``, not at the URI's revision: a build is
+    withdrawn after the commit that published it.
+    """
+    release, build = manifest.release, release_build_id(manifest.build)
+    if release is None or build is None:
+        return
+    url = https_releases_record_url(parts)
+    response = requests.get(url, headers=headers, timeout=HF_REQUEST_TIMEOUT_SECONDS)
+    if response.status_code != 200:
+        raise CertificationError(
+            f"Could not fetch {RELEASES_RECORD_NAME} ({response.status_code}) "
+            f"from {url}; a build in its own release folder is certified only "
+            "while that record lists it."
+        )
+    try:
+        check_build_listed(response.content, release=release, build=build)
+    except ReleaseBuildFolderError as exc:
+        raise CertificationError(str(exc)) from exc
+
+
 def fetch_release_manifest(
     manifest_uri: str,
     token: Optional[str] = None,
 ) -> tuple[DataReleaseManifest, str, dict]:
     """Fetch and parse the data release manifest; returns
-    (manifest, sha256 of the fetched bytes, parsed URI parts)."""
+    (manifest, sha256 of the fetched bytes, parsed URI parts).
+
+    A build published in its own release folder must be read from that
+    folder, and its own files are pinned inside it at the URI's revision
+    (``read_build_folder``). The repository's ``releases.json`` at ``main``
+    must list the build and not have withdrawn it, and must be a record the
+    release writer accepts (``check_build_listed``)."""
     parts = parse_manifest_uri(manifest_uri)
     url = https_manifest_url(parts)
     headers = {"Authorization": f"Bearer {token}"} if token else {}
@@ -257,6 +310,17 @@ def fetch_release_manifest(
     sha256 = hashlib.sha256(response.content).hexdigest()
     payload = json.loads(response.content)
     manifest = DataReleaseManifest.model_validate(payload)
+    try:
+        manifest = read_build_folder(
+            manifest,
+            repo_id=parts["repo_id"],
+            manifest_path=parts["path"],
+            revision=parts["revision"],
+        )
+    except ReleaseBuildFolderError as exc:
+        raise CertificationError(str(exc)) from exc
+    if manifest.build_folder is not None:
+        _require_listed_build(manifest, parts, headers)
     return manifest, sha256, parts