Repository navigation
test: make the heartbeat recovery test independent of write speed - #101
Merged
Merged
Conversation
The test set a 0.1 second alive threshold and ran the recovery pass right after the 8th heartbeat. One slow heartbeat commit on a busy CI runner made the stored heartbeat older than the threshold, so the pass retired the running effect as designed, and the test read "completed". It failed three times on MySQL in the last 30 failed CI runs. The test now waits until the stored heartbeat is more than one threshold past the claim, and runs the recovery pass with its clock set half a threshold after that heartbeat. A slow write can only make the test slower. A new case adds a 0.15 second pause after each heartbeat write: it failed with the CI message before this change and passes after it. Both cases still fail when the heartbeat thread is off.
[Low impact] The test-only change appears safe to merge. SummaryMakes the heartbeat recovery test independent of database write speed.
Reviews (1) · Last reviewed commit: "test: make the heartbeat recovery test i..." · Reviewed by Greptile |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EffectRecoveryTest#test_a_running_effect_keeps_its_process_heartbeat_freshfailed three times on MySQL (trilogy and mysql2) in the last 30 failed CI runs, withExpected: "processing",Actual: "completed".process_alive_threshold = 0.1and runs the recovery pass right after the 8th heartbeat. One slow heartbeat commit makes the stored heartbeat older than 0.1 s, so the pass retires the effect, which is the designed behavior for a stale owner (docs/effect-recovery.md). With the defaults (15 s interval, 60 s threshold), a heartbeat must stall for about 45 s.database_clock_nowset half a threshold after that heartbeat. The heartbeat thread and the database writes stay real.Timeout::ErrorwhenProcessHeartbeat#startis disabled, so they still catch a missing heartbeat.Checks
bundle exec rakepasses on SQLite (984 runs, 0 failures).