Skip to content

A Wikidata corpus whose query starts matching fewer people still builds: carry the short-pool guard to both builders #696

Closed

Description

@derek73

What happens today. build_wikidata_native_corpus.py (#693) stops when a stratum's pool comes back shorter than its size, unless the stratum is marked as its whole population. It also stops when a stratum so marked fills its pool, since the mark is then stale. build_wikidata_corpus.py (#689) has no such check: its pool() accepts whatever WDQS returns. If a query starts matching fewer people (a property is deprecated, a country item is merged), the corpus is rebuilt from a smaller, different sample. The build fails only once a stratum drops below its floor, and its floors sit far under its pools: each pool is six times the stratum's quota, and the floor is the quota.

Why the duplication is part of it. The native builder copies three pieces of its parent:

  • pool()'s result parsing. The two copies already differ by exactly this guard.
  • build()'s stratum loop: dedupe across strata, the missing-person stop, the floor stop.
  • The fetch closure. The LC fetch in the native builder is a fourth copy of the Request/User-Agent/sleep closure, after the Wikidata, GND and CLDR builders.

Proposal:

  1. Move the pool fetch and both size checks into one function in build_wikidata_corpus.py, taking the query text, the pool size and a whole flag. Both builders call it.
  2. Re-pool the parent's strata to measure which are whole populations, and mark those. Malaysia's P5056 stratum matched 17 people on 2026-10-16 (its README entry), so it is one.
  3. Optionally, a shared stratum driver for both build() loops, and a corpus_build.fetch_bytes() for the four fetch closures. The tests' time.sleep patches move with it.

Done when both Wikidata builders stop on a pool of the wrong size, the parent's strata are marked from a measured re-pool, and the offline rebuild of both corpora is byte-identical to the committed files.

Metadata

Metadata

Assignees

Labels

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions