You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A Wikidata corpus whose query starts matching fewer people still builds: carry the short-pool guard to both builders #696
What happens today.build_wikidata_native_corpus.py (#693) stops when a stratum's pool comes back shorter than its size, unless the stratum is marked as its whole population. It also stops when a stratum so marked fills its pool, since the mark is then stale. build_wikidata_corpus.py (#689) has no such check: its pool() accepts whatever WDQS returns. If a query starts matching fewer people (a property is deprecated, a country item is merged), the corpus is rebuilt from a smaller, different sample. The build fails only once a stratum drops below its floor, and its floors sit far under its pools: each pool is six times the stratum's quota, and the floor is the quota.
Why the duplication is part of it. The native builder copies three pieces of its parent:
pool()'s result parsing. The two copies already differ by exactly this guard.
build()'s stratum loop: dedupe across strata, the missing-person stop, the floor stop.
The fetch closure. The LC fetch in the native builder is a fourth copy of the Request/User-Agent/sleep closure, after the Wikidata, GND and CLDR builders.
Proposal:
Move the pool fetch and both size checks into one function in build_wikidata_corpus.py, taking the query text, the pool size and a whole flag. Both builders call it.
Re-pool the parent's strata to measure which are whole populations, and mark those. Malaysia's P5056 stratum matched 17 people on 2026-10-16 (its README entry), so it is one.
Optionally, a shared stratum driver for both build() loops, and a corpus_build.fetch_bytes() for the four fetch closures. The tests' time.sleep patches move with it.
Done when both Wikidata builders stop on a pool of the wrong size, the parent's strata are marked from a measured re-pool, and the offline rebuild of both corpora is byte-identical to the committed files.
Activity
Address review of the short-pool guard: pin the whole marks and the L…
Tighten the LIMIT pin, drop the whole-marks pin, fix prose from the s…
Carry the short-pool guard to the Wikidata corpus builder (#697)
Address review of the shared fetch and stratum loop (#696)
Share the fetch and the stratum loop across the corpus builders (#699)