Add a Wikidata corpus: real second surnames and patronymics, token-aligned

[derek73]

Summary

CLDR (#654) and GND (#688) gave the harness ground truth, but neither can test the two output fields 3.0 is weighing. GND holds García Márquez as one surname and puts a patronymic inside the forename, and CLDR's names are fictional. Wikidata's P1950 (second family name in a Spanish name) and P5056 (patronym or matronym) are the only open labels for those fields that the 2026-10-16 corpus survey found. Wikidata's structured data "is released into the public domain" (CC0).

What's in it

• build_wikidata_corpus.py runs one WDQS query per stratum (one country of citizenship, one property), sequentially, with a descriptive bot User-Agent. It keeps public figures only (deceased, or at least 10 sitelinks), orders them by an MD5 of the item URI so the sample is deterministic, and fetches the items with wbgetentities in batches of 50. Raw responses are cached outside the tree ($NAMEPARSER_TOOLS_CACHE or ~/.cache/nameparser-tools/wikidata). Only name strings and QIDs are committed, with wikidata_LICENSE as the credit.

• Token alignment, not concatenation. Each written name (P1559, and the own-language label) is matched word by word against the stated parts' spellings: labels in the text's language, the stratum's language and mul, plus P1705. A text is kept only if every word is claimed by exactly one part. Other rules:

  • A connective (bin, binti, i, y) joins the part after it: bin Hamzah, i Coll.
  • A part matching in two places, or one item stated as two parts, drops the name.
  • A woman's Russian surname may match the feminine form of its item's label (Лядова for Лядов).
  • The stratum's own field must be written, so Pedro Martínez (display form, no second surname) is out.
  • A family-first text with no comma (Russian official order) is tagged shape 4. A comma makes it a sorting form.

  Stratum	Names	Renderings fully agreeing
  surname2: Spain / Mexico / Colombia	150 / 100 / 80	46/150, 8/100, 1/80
  patronymic: Russia / USSR / Russian Empire	110 / 60 / 40	126/126, 62/62, 41/41
  patronymic: Iceland / Malaysia	100 / 9	1/100, 0/9

  Totals: 668 renderings of 649 names. Malaysia is small: only 17 Malaysian public figures with a P5056 statement matched the query, and 9 of them aligned. Its floor is set to 6. Iceland's pool (270 against 600 asked) also limits it, though it still fills its quota.

• The build fails rather than shrinks. It follows the GND review's pattern:

  • every stratum has a floor, and build() raises before writing if one falls short;
  • responses are shape-checked before caching (a Wikidata API error arrives as HTTP 200), written atomically, and a damaged cache file is named;
  • a query matching nothing raises, and so does a pooled person Wikidata reports missing.

  A deleted name-part item only refuses the person stating it, since --refresh could not fix it.

• cldr_agreement.py gains the source (wikidata_agreement.json) and PENDING_ROLES. These are the design choices Derek made in session:

  • surname2 is scored inside family, as CLDR's own merge does.
  • patronymic (a field CLDR lacks) is scored inside given_names. A patronymic is not a family name (the GND/LC convention), so Vigdís Finnbogadóttir expects family ''. CLDR's own is data states Adólfsdóttir as a surname instead; that conflict between the sources is known and accepted.
  • Each field is also scored on its own once ParsedName has an attribute of that name, with no tool change. Whether the merged role then drops it is left to the 3.0 design. PENDING_ROLES maps each field to the role it is scored inside, and a test holds expected_roles to that map. Until then the summary prints how many renderings state each field and where it is scored: CLDR surname2 369, Wikidata surname2 330, patronymic 338.

• The CLDR, GND and locale records do not move. The snapshot gains 668 names, and no existing reading moves.

What the record says (the worklist)

• Spanish and Latin American double surnames read as middle + family: Rafael Viciana Aráez gives family Aráez. This is the surname2 question.
• Icelandic and Malay patronymics read as family: Soffía Karlsdóttir gives family Karlsdóttir, and Mahathir bin Mohamad gives family bin Mohamad. This is the "family may be empty" question.
• Russian patronymics already agree, by landing in middle.
• One stray finding: Francisco Hernández Pla reads Pla as a suffix.

Blind to (also in the docstring and the README row)

Everything the strict filter drops. The survivors are tidy names whose every word has an item. Short display forms, nicknames and names with an unrecorded part are missing, only eight citizenships are sampled, and no text carries a title, credential or suffix.

Test plan

• uv run pytest: 19,242 passed

• uv run mypy, uv run ruff check

• tests/v2/test_wikidata_corpus.py pins the builder offline on hand-written items. A mutation pass showed 12 decisions unpinned, and each now has a test:

  • alignment: the widest spelling of a part, a comma that may only end a part, connectives (capital I is not one; Malay Bin is);
  • shared items, the feminine forms (women only, including a surname with no masculine ending), P1705-only spellings;
  • required given and first surname, one rendering per order with the native name winning, family-first texts.

  It also stubs the build (quota cut, one stratum per person, the short-stratum refusal) and the network (cache reuse, an error not cached, a damaged cache named, an empty query, missing items).

• Committed-file checks: every field occurs in its text, the fields cover every word of it, and each stratum holds floor ≤ names ≤ quota with its own field written and one stratum per person.

• tests/v2/test_cldr_agreement.py adds Wikidata to the per-source ratchet. It pins the pending-field rules: merged now, and scored on their own once a parsed name answers them.

• Three review rounds, the last over the fix commits. They found:

  • docs promising the merge would end, and a wrong country count;
  • silent shrinkage (no floors, unchecked cache), the pending-role gate skipping more than pending roles, and an attribution stretched from GND to Wikidata;
  • in the fix itself, a missing-item raise that one deleted name item could turn into a permanently broken build.

  All are fixed. Each new guard was mutated and fails, while a harmless control edit passes.

• A /simplify pass:

  • the pending merge is derived from PENDING_ROLES;
  • align() is split into span helpers;
  • the urlopen test stub is shared with the GND tests, and the field-occurs check is one test over every stated-field source.

  The corpus and every record are byte-identical, and the guards were re-mutated and still fail.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (fccaec8) to head (0d3ed46).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #689   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4541     4541           
=======================================
  Hits         4497     4497           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.