Add a native-script Wikidata corpus: real names in their own scripts

[derek73]

Every real-name corpus in tools/differential/ was Latin script, apart from Wikidata's Russian patronymics. CLDR covers native scripts, but its names are invented. So the parser's script-conditional readings had never been checked against a real name: family-first order for Han, Hangul and kana, and Korean surname segmentation. No text holds an honorific, so the corpus does not exercise the CJK honorific peel; it only shows the peel leaves plain names alone. This PR adds 910 real names written in 16 scripts and languages, sampled in 20 strata by country and language.

What it adds

• build_wikidata_native_corpus.py reuses build_wikidata_corpus.py's fetches and token alignment through corpus_build.py, and adds:
  • A script filter. Every letter and combining mark of a text must be in the stratum's script. Wikidata's mul labels are Latin, so without this Anushka Sharma passed as a Hindi name.
  • Character alignment for unspaced Han, kana and Hangul, with the same exactly-one-part rule. Nothing is inferred: a Chinese item that states only P734 is left out, rather than reading the rest of the text as the given name.
  • Feminine surname forms per language (Greek, Bulgarian, Ukrainian, Kazakh), so the filter does not drop women.
  • No shape where the script decides the order, because that order is what the corpus tests. Pure katakana keeps the shape of the order it is written in, matching the parser.
  • A floor per stratum. Below its floor the build stops before writing. A pool that comes back short also stops it, unless the stratum is marked as its whole population (Thailand, Kazakhstan, Bangladesh), and a stratum so marked whose pool fills stops it too.
  • --lc-check, which compares each split with the native-script variants of the person's Library of Congress record. It is a report only: it runs after the build, and adds nothing to the output files and filters nothing.
• cldr_agreement.py scores the source into wikidata_native_agreement.json. Its locale view now maps each source's rendering key to a pack (NATIVE_LABELS for the native strata), so real Han and kana names are scored under ZH and JA too, into wikidata_native_agreement_locales.json.

What it found (2026-10-16)

• Spaced scripts: every one agrees in every role except:
  • seven Arabic two-word surnames (عبد العزيز, عز الدين, شكر الله, ذو الغنى), where the parser reads the first word as a middle name;
  • one Persian double surname;
  • one Bulgarian name whose two stated family names include what is probably the father's name.
• Hangul: 120/120.
• Han and kana, default parser: every unspaced Chinese name disagrees, and 49 of 60 Japanese names.
• Han and kana, locale view: 74/80 under ZH, which misses the surnames 詹, 卫, 柯, 茅, 华 and 文. 59/60 under JA.
• LC cross-check: 99 names have an LC variant that splits them, and 96 of those agree. The other three are Arabic names whose LC record names a person written in the opposite order, so the P244 link is suspect rather than the split. LC writes most Han variants unspaced, so it confirms almost no Chinese or Japanese split.

Wikidata's labels encode a convention in places, and the README records where: an Arabic nisba is entered as P734.

Left out

• Urdu and Marathi: 8 and 5 of 600 aligned.
• The thin strata: Devanagari, Bengali and Thai items are mostly labelled in Latin only, so those strata hold only 10 to 25 names.

Cost

Measured locally without coverage, 2026-10-16:

• the 907 corpus lines (904 names new to the snapshot) add about 2.9 s to the corpus-reading test modules;
• the agreement tests add about 0.6 s.

Records

The snapshot gains 904 names and moves none. The CLDR, GND, Wikidata, conventions and CLDR locale records do not move. The JA rows were recorded with the ja extra installed. The suite passes both with and without namedivider.

Reviews

• First round (correctness, silent failures with 27 mutations): two low-severity bugs, two unguarded paths, and eight untested guards, all fixed in 72a845c.
• Second round, on the fix: one vacuous test row and a stale comment, fixed in 3509acf.
• CI fix: on Python 3.14+, an unclosed HTTPError fails the run; the 404 path now closes it (2bfd5cd).
• Full review (code, comments, tests, types): no correctness bug; the honorific-peel claim corrected, prose made accurate, a stale-whole guard added, and tests for every mutation a second 35-mutation pass left alive (b54bb32).
• /simplify (reuse, simplification, efficiency, altitude): the locale view reads each source's pack from its own rendering key (NATIVE_LABELS beside LOCALE_FILES), so the fields rows no longer carry pack; one shared alignment rule (claim) and one order rule (script_orders, pinned against Policy().script_orders); the agreement tests share one summary run (3.6 s, from 4.6 s). Its second review found a new zh/ja stratum could go unscored; a test now ties the two tables (428148f).
• Negative controls: each new guard was mutated in turn, and each mutation fails a test. The offline rebuild from the cache is byte-identical to the committed files.

Follow-ups (not filed; drafts on request)

• Arabic two-word surnames: should عبد and similar first words bind to the word after them in native script?
• ZH surname coverage: six surnames missed on real names.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (8d7789e) to head (428148f).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #693   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4541     4541           
=======================================
  Hits         4497     4497           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.