Share the corpus builders' cache and file helpers

[derek73]

Summary

This is the follow-up to #689's /simplify pass. Three labeled-corpus builders each carried their own copies of the same helpers:

Helper	Copies before
corpus_lines	CLDR, GND, Wikidata
fields_lines	CLDR, GND, Wikidata
cache_dir	GND, Wikidata
check-before-cache, atomic cached JSON fetch	GND, Wikidata

tools/differential/corpus_build.py now holds them once:

• cache_dir(name)
• parse_json
• cached_json(cache, key, label, fetch, check, refresh)
• corpus_lines
• fields_lines
• write_corpus

Each builder keeps only what is its own: its query, its response predicate and its renderings.

• Import: nothing in tools/differential/ imported a sibling before. Each builder appends its own directory to sys.path before importing. That directory is usually sys.path[0] when a builder runs as a script, but not under -P/PYTHONSAFEPATH, nor when the tests load a builder by path. It is appended rather than inserted, so it cannot shadow a stdlib or installed module. The trade-off: an earlier corpus_build on sys.path would win, and none exists.
• GND's corpus_lines becomes the shape-aware one CLDR and Wikidata used. It writes the same lines for GND's rows, all of which carry "shape": None. Only the test's hand-built rows lacked the key, and they gain it. The shared function stays strict, so a builder that forgets the shape fails rather than emitting untagged lines.
• Cache keys and paths are unchanged, so existing caches are still hit. Wikidata's damaged-cache message now names the URL, as GND's did.
• write_corpus writes the pair through .part files. Both are built and written aside before either is renamed into place, and the .part files are removed on failure. A refused row (a duplicate rendering, a bad shape) or a failed write therefore leaves the committed pair as it was. Only a crash between the two renames could leave them out of step. The old builders wrote the corpus file first, then the fields file.
• Sort order: an untagged line sorts before the tagged ones of its text. The old key tied shape 0 with None, leaving that order to set iteration. No builder writes shape 0, so no output moves.
• README: "Adding a source" now points new builders at corpus_build.py, and the Wikidata cache path names its wikidata/ directory.

Test plan

• All three corpora rebuild byte-identical: CLDR re-fetched from GitHub, GND and Wikidata from the cache.

• uv run pytest: 19,251 passed

• New tests/v2/test_corpus_build.py pins each helper's contract:

  • each builder loads in a fresh -P interpreter. In the test process, the first builder loaded had been finding the module for the other two, so a builder that forgot its own sys.path step still passed;
  • the shape TypeError;
  • the untagged-first order, over forty texts, since one text passed the old key by luck;
  • write_corpus: a refused row, and a write that fails partway, each leave the old pair and no .part file;
  • the atomic cache write, by fault injection.

  Each test was checked against a mutation of its code and fails.

• uv run mypy, uv run ruff check

• Three review rounds:

  • the first found no behavior change;
  • the second (tests, comments, silent failures) led to the fixes above;
  • the third, over the fix commits, found that "both or neither" held only for a refused row, which the .part writes now extend to a failed write.

  The deliberate behavior changes don't move output: the pair is never half-written, and the tie-break is deterministic.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (4375b01) to head (beac25d).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #690   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4541     4541           
=======================================
  Hits         4497     4497           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.