Add a GND corpus: real names with their parts stated, scored like CLDR

AGENTS.md

@@ -261,7 +261,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_
 
 **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`.
 
-**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672); its `JA` rows run only with the `ja` extra installed.
+**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672); its `JA` rows run only with the `ja` extra installed.
 
 ## Gotchas
 

tests/v2/test_cldr_agreement.py

@@ -1,11 +1,12 @@
-"""The CLDR agreement ratchet, and the comparison rules it rests on.
+"""The CLDR and GND agreement ratchets, and the comparison rules they rest on.
 
-tools/differential/cldr_agreement.py decides, for each CLDR rendering and
-each role, whether nameparser agrees with CLDR's stated fields. The rules
+tools/differential/cldr_agreement.py decides, for each rendering and
+each role, whether nameparser agrees with the source's stated fields
+(CLDR's, and the GND's written in CLDR's vocabulary). The rules
 are pinned here first, each with the wrong answer the plausible
 alternative gives recorded beside it; the ratchet test at the end then
-holds the whole corpus to the recorded disagreements in
-tools/differential/cldr_agreement.json.
+holds each source's corpus to its recorded disagreements in
+tools/differential/cldr_agreement.json and gnd_agreement.json.
 
 tools/ is not a package, so the module is loaded by path, as
 _differential_fixtures.py loads compare.py.
@@ -152,26 +153,41 @@ def test_changes_reports_a_rendering_that_is_gone() -> None:
     assert message.startswith("gone: b/nativeG/givenFirst")
 
 
-def test_the_corpus_and_the_fields_file_hold_the_same_renderings() -> None:
-    from_fields = {(row["text"], row["shape"]) for row in agreement.load_rows()}
+#: Each source's corpus file, and a floor its distinct texts must
+#: exceed: a truncated file would otherwise pass both checks below, and
+#: an emptied one re-recorded would match its emptied record.
+_CORPORA = {"CLDR": ("corpus_cldr.jsonl", 1000), "GND": ("corpus_gnd.jsonl", 500)}
+
+
+def test_every_source_has_a_corpus_file() -> None:
+    assert set(_CORPORA) == set(agreement.sources())
+
+
+@pytest.mark.parametrize("source", sorted(_CORPORA))
+def test_the_corpus_and_the_fields_file_hold_the_same_renderings(source: str) -> None:
+    fields, _ = agreement.sources()[source]
+    corpus, least = _CORPORA[source]
+    from_fields = {(row["text"], row["shape"]) for row in agreement.load_rows(fields)}
     from_corpus = set()
-    for line in (_TOOLS / "corpus_cldr.jsonl").read_text(encoding="utf-8").splitlines():
+    for line in (_TOOLS / corpus).read_text(encoding="utf-8").splitlines():
         if line.strip():
             entry = json.loads(line)
             from_corpus.add((entry, None) if isinstance(entry, str)
                             else (entry["name"], entry["shape"]))
     assert from_corpus == from_fields
-    assert len(from_fields) > 1000, "the CLDR corpus is missing or truncated"
+    assert len(from_fields) > least, f"the {source} corpus is missing or truncated"
 
 
-def test_nameparser_still_agrees_with_cldr_where_it_did() -> None:
-    rows = agreement.load_rows()
+@pytest.mark.parametrize("source", sorted(_CORPORA))
+def test_nameparser_still_agrees_with_each_source_where_it_did(source: str) -> None:
+    fields, record = agreement.sources()[source]
+    rows = agreement.load_rows(fields)
     problems = agreement.changes(
-        agreement.read_record(), agreement.disagreements(rows),
+        agreement.read_record(record), agreement.disagreements(rows),
         {agreement.rendering_key(row) for row in rows})
     assert not problems, "\n".join([
         *problems[:40],
-        f"{len(problems)} change(s) against tools/differential/cldr_agreement.json.",
+        f"{len(problems)} change(s) against tools/differential/{record.name}.",
         f"If every one is intended, re-record with `{agreement.RECORD_COMMAND}` "
         "and say why in the commit."])
 
@@ -183,7 +199,9 @@ def test_the_summary_counts_each_role_and_matches_the_record(
     for role in agreement.ROLES:
         assert any(line.split()[0] == role and "agree" in line
                    for line in out.splitlines() if line.strip()), role
-    assert "matches the record" in out
+    for source in agreement.sources():
+        assert f"{source} view matches the record" in out
+    assert "locale view matches the record" in out
 
 
 @pytest.mark.parametrize("text", [
@@ -253,12 +271,20 @@ def test_a_re_record_drops_entries_whose_file_left_the_table(
         tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
     # The carry-over that keeps an unmeasured pack's entries must not
     # keep an unmapped file's too.
+    # Every record is redirected through sources() itself, so a source
+    # added later cannot be written into the tree by this test.
     record = tmp_path / "locales.json"
     agreement.write_record({"xx/nativeG/sorting": {"text": "X"}}, record)
+    real = agreement.sources()
+    redirected = {name: (fields, tmp_path / f"{name}.json")
+                  for name, (fields, _) in real.items()}
     monkeypatch.setattr(agreement, "LOCALE_RECORD", record)
-    monkeypatch.setattr(agreement, "RECORD", tmp_path / "default.json")
+    monkeypatch.setattr(agreement, "sources", lambda: redirected)
     assert agreement.main(["--record"]) == 0
     assert "xx/nativeG/sorting" not in agreement.read_record(record)
+    # and the re-record wrote every source's record, not CLDR's alone
+    for name, (fields, written) in redirected.items():
+        assert agreement.read_record(written) == agreement.read_record(real[name][1]), name
 
 
 @pytest.mark.parametrize("pack", [