Skip to content

State the conventions corpus's parts, and score them like CLDR's - #692

Merged
derek73 merged 3 commits into
masterfrom
feat/conventions-fields
Oct 17, 2026
Merged

derek73 merged 3 commits into
masterfrom
feat/conventions-fields

Conversation

@derek73

@derek73 derek73 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #686 / #691: the conventions corpus gets a fields file after all, so a change that moves one of its names shows whether the reading moved toward the expected one or away from it.

What.

  • 30 of conventions.py's 35 rows carry fields in CLDR's vocabulary, beside their expect note.
  • The other 5 carry an unstated reason. The builder refuses a row with neither or both, so a row can't lose its fields unseen.
  • build_conventions_corpus.py writes conventions_fields.jsonl.
  • cldr_agreement.py scores it as a fourth source, recorded in conventions_agreement.json.
  • The scorer gains one field, honorific, scored inside suffix.

Where the readings come from. The research the rows were written from: the 2006 UK naming-practices guide and the Cultural Atlas report, both in the gitignored docs/superpowers/, read back from the sessions that wrote #686. The fields follow the Wikidata source's conventions: a patronymic is written with its connective (bin Yusof) and scored inside given_names, and a surname2 inside family.

Your calls.

  • An honorific written after the name (Bey, ssi, san, -san, úr) is a suffix. That matches the design notes and nameparser's reading of さん; CLDR's さん-as-title disagreement stays recorded.
  • Kaur is given2.
  • Péterné is the given name.
  • H. Budi Santoso is title + one personal name.
  • Ma.'s Cruz is a surname2.
  • Vda. de Mora sits inside the surname.
  • wa is left unstated beside arap/ole.

Left unstated (still in the corpus, still watched by the snapshot), since the scorer reads an unstated part as an empty role:

  • the kunya Abu Karim
  • Szabóné Kiss Éva
  • arap/ole/wa, whose last word is a parent's name or a family name depending on the family

Guards added in review, since this source is hand-written:

  • A field name no rule reads now raises, for every source; CLDR's locale and given-informal are skipped on purpose. A mistyped honorfic had been passing as agreement.
  • test_the_fields_partition_their_rendering: every word of a non-CLDR rendering is stated by exactly one field. GND, Wikidata and conventions already held it.
  • The scorer has a FAMILY_FIRST_GIVEN_LAST parser, so such a row no longer crashes it.

Measured.

  • 5 of 30 rows agree in every role: Harpreet Kaur Dhillon, Szabó Péterné, Kim Min Ho, Ivan Petrov Dimitrov, Lim Mei Ling Jessica.
  • All 25 disagreements are real misreadings. The largest group is a patronymic misplaced, usually read as the family name (Hassan bin Yusof → family bin Yusof), which makes the record a worklist for the 3.0 patronymic field. Titles read as given names (U Kyaw Win) and honorifics read as name words are the next groups.
  • Re-recording leaves the CLDR, GND, Wikidata and locale records unchanged, and the corpus file is byte-identical.
  • Each new guard was checked against a bad row and fails on it.
  • Full suite: 19,330 passed; mypy and ruff clean.

🤖 Generated with Claude Code

Follow-up to #686 (#691). 30 of conventions.py's 35 rows now carry
`fields` in CLDR's vocabulary, drawn from the UK naming-practices guide
and Cultural Atlas research the rows were written from;
build_conventions_corpus.py writes them to conventions_fields.jsonl and
cldr_agreement.py scores them as a fourth source, recorded in
conventions_agreement.json: 5 of 30 agree in every role.

Derek's calls: a word of address written after the name (Bey, ssi,
san) is a new `honorific` field scored inside `suffix`, as nameparser
reads さん; Kaur is given2, Péterné the given name, H. Budi Santoso a
title and one personal name, Ma.'s Cruz a surname2 and Vda. de Mora
part of the surname. Five rows stay unstated, since a stated field
reads an unstated part as an empty role: the kunya, the glued -san,
Szabóné's married form, and arap/ole, patronymic or family name by
family. The fields file is therefore a subset of its corpus, which
test_cldr_agreement now allows for this source alone.

The other sources' records do not move; the corpus file is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 2.5 milestone Oct 17, 2026
@derek73 derek73 self-assigned this Oct 17, 2026
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.03%. Comparing base (c89629b) to head (4876318).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #692   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files          46       46           
  Lines        4541     4541           
=======================================
  Hits         4497     4497           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

derek73 and others added 2 commits October 16, 2026 20:10
…the text

Review of #692:
- A field name no rule reads now raises in expected_roles, for every
  source: a mistyped honorfic was expected in no role and passed as
  agreement. CLDR's locale and given-informal are the only names
  skipped on purpose (KNOWN_FIELDS, IGNORED_FIELDS).
- test_the_fields_partition_their_rendering: every word of a non-CLDR
  rendering is stated by exactly one field, so a left-out part fails.
  GND, Wikidata and conventions already held it.
- Each conventions row carries fields or an unstated reason, and the
  builder refuses neither or both, so a row's fields cannot drop unseen
  behind the subset check.
- _PARSERS gains shape 5, so a FAMILY_FIRST_GIVEN_LAST row no longer
  crashes the scorer.
- Suzuki-san is stated after all: unspaced text matches by substring,
  so surname Suzuki + honorific -san partition it. My earlier reason for
  leaving it out was wrong. 26 of 31 rows now disagree.
- Stale docstrings (two sources named of four), the ur note (a word of
  address, as its fields say), and the record of where today's reading
  lives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Derek's call: the research groups the three as declining Kenyan
parent-name markers whose last word may now be a family name, and its
wa example names a mother, so wa is no surer a patronymic than the
others. Its row stays in the corpus; 30 rows are stated, 25 disagree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 8d7789e into master Oct 17, 2026
12 checks passed
@derek73
derek73 deleted the feat/conventions-fields branch October 17, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant