Repository navigation
Conversation
schema.sql stays the schema as it is now. A change to an existing cluster goes in a numbered file in migrations/, applied once, in order, on one node. 000_baseline.sql is schema.sql as of this PR and is never edited; the README says how to run, document and record a migration. schema.sql's header points to it.
test/check.sh starts a one-node oonidata_cluster with Keeper at production's version (24.8.6.70), loads schema.sql into one database and every migration, in order, into another, and compares each table's definition and Keeper path with the database name and UUIDs blanked. It takes about 10 seconds. CI runs it when schema.sql, the migrations, the test or the workflow change. It fails when a migration is missing, and when schema.sql changes without one: both checked by hand before this commit.
The view kept its rows in a hidden table, .inner_id.<uuid>, sorted by
(probe_cc, probe_asn, input). It's a SummingMergeTree, so merges add up the
weeks of each key and keep one arbitrary week. Its reader, ooniprobe's
test-list/urls (common/prio.py), sums the current and previous week, so it
misses every count whose merged row says an older week, which in production
is most of them. Tested on 24.8: 2 rows from last week and 5 from this one
merge into one row of 7 that says last week.
The view now writes TO counters_asn_test_list_data, sorted by (probe_cc,
probe_asn, input, week), with a plain {database} Keeper path. It keeps its
name and column types, so the reader's query doesn't change. The table is
backed up and restored by name, like any other table, by the
clickhouse_backup role's "*" job; the view's backup holds only its
definition. Restore the table before the view: a view restored without its
table makes every insert into fastpath fail.
Migration 001 does it in production. The old rows are wrong, so it doesn't
copy them: it counts the two weeks the reader uses again from fastpath, from
the rows inserted before the new view existed (fastpath's update_time), and
the new view counts the rest. That also counts uploads that came more than
8 days late, which the view leaves out. It isn't a mutation. Queries on the
view fail between the DROP and the RENAME, for about a second; from then on
test-list/urls gets the corrected counts.
Tested on a two-node 24.8.6.70 cluster set up like production (the view on
one node, its rows already merged): the weeks come out right, a late upload
is counted once, later inserts are counted once on both nodes, and
test/check.sh passes.
aagbsn
marked this pull request as draft
October 23, 2026 11:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a base schema (current state), a migrations directory for changes to prod, and a script that ensures they stay in sync. schema.sql will be used for ooni/backend's CI fixtures, so that there is a single source of truth for database tables. Related is #520, for backing up (soon, also restoring from) clickhouse backups. For handling of some tables, e.g. the EmbeddedRocksDB types, this is convoluted, so migrating them to replicated types is desirable.