Repository navigation
Conversation
Nothing backed up the production cluster: the weekly ooni-db-backup export reads a legacy local ClickHouse (its fastpath files have been empty since mid-2025), and scripts/dump-tables-ch.sh listed tables that no longer exist while missing obs_*, analysis_web_measurement and others. Add a clickhouse_backup role that runs ClickHouse's native BACKUP daily from a systemd timer. Each table has its own chain: a full backup every 30 days and incrementals in between, so a failure in obs_web (3.9 TB) doesn't cost the other tables their backup. The two newest complete chains are kept; chains pinned with --keep (e.g. before an upgrade step) are kept until released. A host runs the jobs set in its host_vars, one per bucket: - data3, "measurements": the replicated tables, minus the private ones and data1's. data3 isn't behind clickhouseproxy and doesn't run Airflow. - data3, "private": msmt_feedback, faulty_measurements, url_priorities, in their own bucket, since R2 tokens grant access per bucket. - data1, "data1": the plain MergeTree tables whose data only data1 holds (labeling_frames, event_detector_cusums, event_detector_v2_changepoints, changepoint_label). Remove a table from clickhouse_backup_data1_tables once it's replicated and data3 picks it up. The R2 endpoint and token come from SSM. R2 rejects multipart uploads whose parts (all but the last) differ in size, so the uploads use fixed 128 MiB parts (s3_upload_part_size_multiply_factor = 1). With R2's 10,000-part limit a file can reach 1.25 TiB; the largest file of a backup is one column of one part, and parts stop growing at 150 GB. After each table the script writes its result to a node_exporter textfile (ooni_clickhouse_backup_*), and prometheus_node_exporter gets a node_exporter_textfile_dir option for it. New alerts: a table's backup failed, no backup of a table for 3 days, no results from data1 or data3 for a day, and data3's load over half its CPUs for 15 minutes. TextfileJam skips the backup file, which changes daily rather than hourly. Things found while testing against 24.8.6.70 and MinIO: - BACKUP saves only the schema of EmbeddedRocksDB and Join tables, so their rows are exported in Native format alongside the backup. - An incremental backup stores its base_backup argument, keys included, in its .backup file. ClickHouse gets the bucket keys from <s3> endpoints in config.d instead, which it reloads without a restart. - max_backup_bandwidth is a query setting (BACKUP ... SETTINGS rejects it), and actual throughput is about a third of the value. - A 24.8 server segfaults restoring a backup taken on 26.8, so the runbook pins a backup before each upgrade step. Remove the outdated dump and restore scripts.
A table started a new chain once its chain was 30 days old, counted from when its full began. With weekly runs that gives a full every 5 weeks, and a threshold of 28 days would slip a week whenever a table's backup started a few minutes earlier than it did in the full. A table now starts a new chain at its first backup in each calendar month (UTC): the 1st with daily runs, the first Sunday with weekly ones. A full that fails is retried at the next run, as the latest complete chain is still from the month before. clickhouse_backup_full_interval_days is gone.
Requires= in the timer unit started ooni-clickhouse-backup.service whenever the timer started: at every boot of data1 and data3, and on every deploy that restarts the timer. Backups now start only at OnCalendar; Persistent=true still runs one missed while the host was down.
The alerts only said when something went wrong. At the end of a run the script now records each job's result (ooni_clickhouse_backup_last_ run_*: when it finished, how long it took, tables backed up, in full, and failed), and ClickHouseBackupCompleted fires for an hour after a run with no failed table. It goes to a new Alertmanager receiver, slack-notice: one message per job run in #ooni-bots, without the [FIRING] prefix, a "resolved" message or email. ClickHouseBackupFailed now says that nothing was deleted and that the next run retries the table.
"1 tables backed up, 1 of them in full" now reads "Tables backed up: 1, in full: 1".
… it changes tests/run.sh renders the role's configs from its templates, and Alertmanager's from its role, with Ansible, then: - promtool unit-tests the backup alerts in alert_rules.yml, and amtool checks Alertmanager's config and that the alerts reach the right receivers; - test_schedule.py checks when full backups start and what is pruned, over months of daily and weekly runs with failures; - test_e2e.py runs the script against ClickHouse 24.8.6.70, MinIO for R2, node_exporter, Prometheus 2.42.0 and Alertmanager 0.25.0 (the versions in production) and a stand-in for Slack, in docker compose: full and incremental backups and a restore; a failed run; a month's new full interrupted by a ClickHouse restart, which deletes nothing, then completed with the oldest chain pruned; and the Slack messages and emails each causes. The workflow runs it on pull requests and pushes to main that change this role, the alert rules or Alertmanager's role. About 7 minutes.
The script now runs on Debian bookworm with its python3 and python3-boto3, the packages the role installs on the ClickHouse hosts, instead of a Python image with boto3 from pip. The workflow installs ansible-core and python3-boto3 with apt for the scripts around it.
…ured max_backup_bandwidth was 120 MB/s, on the strength of 24.8 uploading at about a third of it in tests against MinIO. If production doesn't show the same, that would take most of the nodes' 1 Gbit/s link. Cap it at 50 MB/s, about 31 hours for a full backup of everything, and raise it if the first backups show the third: about 4 days for a full, longer than ClickHouseBackupStale's 3. The comments and the runbook say how to check the rate.
counters_test_list is detached permanently, so it's no longer in system.tables, which the script takes its tables from.
The runbook listed fingerprints_* and session_expunge among the tables whose rows each backup exports, but neither is backed up: the fingerprints tables are only on data1 and data2 and the hourly updater rebuilds them, and session_expunge is empty on both. Only test_groups, on data3, is exported.
The experiment table on data2 is gone, and data2 isn't backed up anyway.
The listing is the command's output, for stdout and pipes; log is for what run and release report, to stderr and the journal.
A full backup of everything takes more than a day, so run weekly instead of daily for now: incrementals every Sunday at 02:30 UTC, each on top of the one before, and a new full on the first Sunday of each month. The timer's first run after a deploy is then the next Sunday; the runbook says how to start the first full on deploy day.
…upStale With weekly runs a table's last backup is normally up to a week old, so ClickHouseBackupStale fired after 3 days would fire every week. It now fires after 8 days, and not while the table's job is still running: on the first Sunday of a month the fulls take more than a day, and the largest tables come last. The promtool tests cover both.
…run on the host The alert was held back only while the table's own job ran. On data3 the jobs run one after the other, so on the first Sunday of a month private waited more than a day behind measurements' full backups and passed 8 days without one, while nothing was wrong. And a run killed without reaching its finally left running at 1, which held the alert back for good. The script now also exports, per job, when its current run started (ooni_clickhouse_backup_running_since_timestamp_seconds, 0 when idle). The alert is held back while any job on the host is in a run that started in the last 5 days. The promtool tests cover a 7- and a 9-day old table with nothing running, a job waiting behind another job's run, and a stuck run; test_job.py the new metric and older state files.
…tables The other jobs ran with no tables, recorded a completed run of 0 tables (ClickHouseBackupCompleted posted it to Slack) and overwrote their real weekly results.
At boot, Persistent=true catches up a missed Sunday before ClickHouse accepts queries. The checks before the table loop then failed outside any try: the run died with a traceback, skipped the later jobs and wrote no metrics, and nothing alerted until ClickHouseBackupStale 8 days later. A run now waits up to 10 minutes for ClickHouse (clickhouse_wait_seconds). If it still can't start, the job's tables are marked failed, so ClickHouseBackupFailed says so, and the next job runs. One failed status poll also gave up on a backup that can run for days, while the BACKUP carried on server-side. Polls are now retried for up to 10 minutes.
A host has no backup results until its first run, which with weekly runs can be up to a week after the role is deployed: the 26 h wait alerted every 3 hours until then.
…restore test_groups Before upgrading, step 4 re-enabled the timer first, which with Persistent=true may start a run for a missed Sunday, then ran a multi-day run --full in a shell. Run it with systemd-run while the timer is still disabled, and re-enable the timer afterwards. Note that ClickHouse waits for running backups when it stops (shutdown_wait_backups_and_restores, TimeoutStopSec=infinity), so a restart during a full hangs until it is done. The restore example loaded fingerprints_dns, which isn't backed up; test_groups is the table it's for.
…, test the backup host alert node_exporter's file label is the file's full path, so the anchored "ooni_clickhouse_backup.*" never matched (TextfileJam matches nothing today anyway: it uses node_exporter's pre-0.16 metric names). The promtool tests now cover the exclusion, and ClickHouseBackupHostLoadHigh, whose description no longer assumes a backup is the cause.
An empty include_tables meant every table. data1's job includes only clickhouse_backup_data1_tables, which is to be emptied as its tables become replicated; data1 would then have backed up every table, private ones included, into the measurements bucket, under the same prefixes as data3, whose in-progress full it could also have removed as incomplete. "*" now selects every table and an empty list none, with a warning; data3's measurements job and the role's default job say ["*"].
…cked up The state file kept every table and job it had ever seen, so a table dropped, or moved to another job (as data1's tables will be once replicated), kept its old results and alerted forever: ClickHouseBackupStale after 8 days, or ClickHouseBackupFailed. After a run without --table, a job now forgets the tables it no longer selects, and every run forgets the jobs no longer in the config.
…letes, fsync state - running_backups() matched the endpoint by substring, so the measurements bucket also matched the private bucket's backups; match <bucket>/<prefix>/. - delete_objects reports keys it didn't delete in a 200 response, which were ignored; raise on them. - Fsync the state and metrics files before they replace the old ones, and start from no results if the state file is unreadable instead of failing every command. - Pass the upload settings to the EmbeddedRocksDB/Join export too: R2 needs its multipart parts to be the same size.
…utput and logs on failure ClickHouse waits for running backups when it stops (shutdown_wait_backups_and_restores), so the interrupted-full scenario only worked because compose restart killed it after 10 s; kill it instead. Failing run checks now show what the script said, run.sh prints the containers' logs when the test fails and removes containers left by KEEP=1 before starting, and the unit tests silence the script's log.
…ocksDB rows, and the new full after pruning Only the newest incremental was restored. Two more runs now give fastpath a full and two incrementals, and each of the three restores fastpath as it was at that backup (200,000, 201,000 and 202,000 rows, by count and a checksum of every row). fingerprints_dns, an EmbeddedRocksDB table, is restored as the runbook says: RESTORE its schema, then load data.native.zst. After the interrupted full and the pruning, the new full restores obs_web. The run takes about 9 minutes.
…vers A table created on data1 or data2 only, like data1's own tables were, was in no backup and nothing said so: data3 doesn't have it, and data1's job lists its tables. After each run, data3 now lists the tables on every node (clickhouse_backup_check_cluster: oonidata_cluster) and reports those that no job here backs up (from a table on this host), that aren't in clickhouse_backup_data1_tables (backed up by data1) and that aren't excluded, as ooni_clickhouse_backup_table_not_backed_up. ClickHouseTableNotBackedUp names the table and its nodes and says how to fix it. clickhouse_backup_exclude_tables becomes the list of tables that are never backed up, each with its reason: fingerprints_dns and fingerprints_http (the hourly updater rebuilds them) and session_expunge (empty, unused) join it. That changes no backup: they're not on data3, and data1's job lists its tables. The end-to-end test's EmbeddedRocksDB table is now test_groups, as in production.
…that no backup covers The test environment gets a second ClickHouse node, data1, with data3 in one test cluster. A new step creates a table on data1 only and checks that ClickHouseTableNotBackedUp names it in Slack, and no other table (labeling_frames is data1's, asnmeta_tmp is excluded). It comes after the Completed step, whose check that nothing went to email it would otherwise break.
aagbsn
marked this pull request as draft
October 22, 2026 15:12
aagbsn
commented
Oct 22, 2026
| clickhouse_backup_private_tables: | ||
| - msmt_feedback # account ids and free-text comments | ||
| - faulty_measurements # until its contents are reviewed | ||
| - url_priorities # until it's decided whether it's published |
Contributor
Author
There was a problem hiding this comment.
these don't actually need to be private
aagbsn
commented
Oct 22, 2026
| clickhouse_backup_data1_tables: | ||
| - labeling_frames | ||
| - event_detector_cusums | ||
| - event_detector_v2_changepoints |
Contributor
Author
There was a problem hiding this comment.
this should be replicated so that backup jobs can run from any cluster node
aagbsn
commented
Oct 22, 2026
|
|
||
| - `ClickHouseBackupFailed`: a table's last attempt failed. Nothing is | ||
| deleted, and the next run retries it. | ||
| - `ClickHouseTableNotBackedUp`: a table on the cluster is in no backup job and |
Contributor
Author
There was a problem hiding this comment.
alerting for tables that weren't backed up, e.g. new tables were created but not added to the backup job
aagbsn
commented
Oct 22, 2026
| ``` | ||
| and the alert rules with `deploy-monitoring.yml`. | ||
| 4. The timer takes the first (full) backup at its next run, on Sunday at | ||
| 02:30 UTC. A full backup of everything takes about 31 hours, or longer if |
Contributor
Author
There was a problem hiding this comment.
because a full backup is expected to take more than 24 hours, maybe these should fall on another day of the week, so that they are completed before monday in case of any new deployments.
aagbsn
commented
Oct 22, 2026
| | data3 | `private` | `ooni-clickhouse-backup-private` | `clickhouse_backup_private_tables`: `msmt_feedback`, `faulty_measurements`, `url_priorities` | | ||
| | data1 | `data1` | `ooni-clickhouse-backup` | `clickhouse_backup_data1_tables`: plain MergeTree tables whose data only data1 holds | | ||
|
|
||
| data3 takes the replicated tables because it isn't behind clickhouseproxy |
Contributor
Author
There was a problem hiding this comment.
clickhouseproxy doesn't have anything to do with where backups should be pulled from, but data3 has the least critical loads
aagbsn
commented
Oct 22, 2026
| so one backup file can reach 1.25 TiB. | ||
| - Two complete chains are kept per table (`clickhouse_backup_keep_chains`), | ||
| so a month or two of restore points. | ||
| - Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A |
Contributor
Author
There was a problem hiding this comment.
The impact of the backup should be measured to determine what appropriate limits are. data3 is not SSD storage and has a 1gbit (unmetered) link; faster capacity is possible but at a cost.
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backup scripts and alerting for backing up to S3 compatible storage. Tests in harness for backup and restore with full and incremental snapshots; adds alerting for failed jobs and missing (e.g. new) tables.