Add a GND corpus: real names with their parts stated, scored like CLDR

[derek73]

Summary

CLDR's person-name data (#654) gave the harness its first ground truth, but its names are fictional. This adds a second labeled source of real people: the German National Library's authority file (GND), read through lobid-gnd.

GND stores each person's preferred name in parts:

• surname, forename, and prefix (the particle);
• counting, a regnal number;
• or a single personalName, for someone entered under the whole name with no surname (Icelanders, Malays, rulers, saints).

A corpus survey on 2026-10-16 ranked GND first among open sources, for three reasons:

• License: CC0 with no conditions. DNB's own page says the GND "is available free of charge for general use under Creative Commons Zero (CC0 1.0) terms".
• Native fields: the name parts are stored separately, so nothing has to be parsed out of a string.
• Surname or not: its personalName/surname split answers whether a person has a family name at all.

What's in it

• build_gnd_corpus.py samples only people who are deceased and have a Wikipedia article, which keeps the corpus to public figures.
  • Size: 801 renderings of 528 names.
  • Index form: each preferred name as cataloged (Gogh, Vincent van).
  • Composed form: outside family-first areas, also a given-first form built from the fields (Vincent van Gogh), flagged composed: true.
  • Sampling: by shape and area, never at random. Each stratum takes a deterministic id slice, in sha256 order.
  • Stops before writing when a stratum falls below its floor, a query matches nothing, or a pool pages short, so a change in lobid fails the build instead of shrinking the corpus.
  • Cache: raw responses are cached outside the tree ($NAMEPARSER_TOOLS_CACHE/gnd), as tools/name_usage.py does. Each cache file is validated on read and written atomically. Only the derived rows are committed, with gnd_LICENSE as the credit.
• cldr_agreement.py now takes more than one source (sources()).
  • gnd_fields.jsonl is written in CLDR's field vocabulary, so the existing comparison rules apply unchanged.
  • GND gets its own record (gnd_agreement.json), held by the same ratchet test, now parametrized per source.
  • --record prints, per source, how many changes of each kind it accepts.
  • The locale view still reads CLDR only.
• Only what GND states is compared.
  • The particle split is compared only where a prefix stands before a one-word surname.
  • Otherwise the family is stated whole: GND does not split a particle inside a surname (Vélez de Guevara, Zuylen van Nijevelt), so a missing split means "not stated", not "no particle".
  • A glued particle (d'Arenberg) is stated whole in its composed line.
  • A test holds that every field occurs in its rendering's text.
• snapshot.jsonl gains 797 names, and no existing reading moves.

Agreement, measured 2026-10-16

Role	Agree
title	800/801
given_names	599/801
family	580/801
suffix	777/801
family_particles	218/259
family_base	258/259

579 of the 801 renderings agree in every role. Per stratum:

Stratum	Renderings agreeing in every role
plain-anglo, plain-europe	50/50, 50/50
index-cjk, index-hu	40/40, 20/20
particle-de-at, nl-be, fr, iberia, it	59/60, 58/60, 47/50, 49/50, 36/40
particle-elided, particle-arab	12/26, 20/40
compound-iberia, compound-latam	42/60, 38/60
regnal	17/40
direct-is, my-id, arab, ir-af, et-mm, eu	0/30, 10/30, 4/30, 3/20, 2/15, 22/30

What the record says (the worklist)

• Icelandic patronymics and Malay names are given a family name. Kristján Eldjárn reads family Eldjárn. This is the "family may be empty" question 3.0 is weighing.
• Spanish and Latin American compound surnames read as middle + family. Josep Domènech Estapà. This bears on surname2.
• Regnal numbers read as a family name. Sancho I. reads family I., while Johann II. reads it as a suffix.
• Elided and Arabic particles are not split off. Arenberg, Auguste Marie Raymond d' and Ǧaʿfarī, Ibrāhīm al-.

What it is blind to (also in the docstring and the README row)

• GND writes names in German library romanization and gives historic figures their German forms. Latin script only.
• Titles, credentials and nicknames are kept out of preferred names, apart from a few inside a personalName, such as Imam Bonjol.
• Cataloging rules decide what counts as a surname, not the bearer's culture. Vietnamese and Thai names are entered as a personalName, so no stratum samples those areas.
• A stratum's count reflects what was sampled, not how common the shape is.

Review rounds

/pr-review-toolkit:review-pr ran the code, test, comment and silent-failure reviewers. The fixes are in 45dc2a8:

• Unstated particles: 19 renderings recorded the parser's de inside a GND surname as a disagreement no fix could clear. Such surnames are now stated whole.
• Loud failures: the build now stops instead of shrinking the corpus, and the cache is validated.
• Guards: the regnal branch checks the preferred name, and a record with an unknown field is refused.
• Tests: each guard is isolated, and pool, build and cache have offline tests. Six mutants of the new guards are each killed.
• Docs: the license quote is now verbatim.

A second review of that fix commit found the same unstated-particle gap one shape over, inside a several-word core after a prefix (Zuylen van Nijevelt, Julius van). That is fixed in 1de7ee0, and its revert is killed by the new test. The README's new "Adding a source" note records both misses for the next source.

Release figures (decided)

A release figure counts GND's moved names in, its composed renderings included, because the names are real (Derek, 2026-10-16). AGENTS.md's rule that counts CLDR's fictional renderings out is unchanged. The decision is recorded in tools/differential/README.md.

Test plan

• uv run pytest: 18,079 passed
• uv run mypy, uv run ruff check
• The negative controls for the new guards each fail their test when reverted (mutated in place, then restored from git).
• The re-record test redirects every source's record through sources(), so it cannot write into the tree, and it checks that every record was rewritten.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (1a4e8a1) to head (fd926d2).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #688   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4541     4541           
=======================================
  Hits         4497     4497           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.