Skip to content

Compactor: delete orphaned global block deletion marks - #7890

Open
cvas22 wants to merge 2 commits into
cortexproject:masterfrom
cvas22:fix/7885-orphaned-deletion-marks
Open

cvas22 wants to merge 2 commits into
cortexproject:masterfrom
cvas22:fix/7885-orphaned-deletion-marks

Conversation

@cvas22

@cvas22 cvas22 commented Oct 21, 2026 •

Copy link
Copy Markdown

What this PR does:

When a block is gone from the bucket but its deletion mark is still in the tenant's markers/ location, the bucket index updater logs skipped missing block deletion mark on every run and never removes the mark. The tenant location then stays in the bucket forever, even with no blocks and no bucket index.

This change deletes such a mark from the updater. To avoid racing with a block that is being marked for deletion (the global mark is uploaded before the per-block one), the mark is only removed when nothing exists under the block's location. If listing or deleting fails, it is logged and the update continues.

Only block deletion marks are handled. Orphaned no-compact marks are not touched.

Tests: new TestUpdater_UpdateIndex_ShouldDeleteOrphanedGlobalDeletionMarks (fails without the change). Ran the bucketindex and compactor blocks cleaner tests on Linux.

Which issue(s) this PR fixes:
Fixes #7885

Checklist

  • Tests updated
  • Documentation added (no user-facing config or documented behavior changes)
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags (no new flags)

@cvas22
cvas22 marked this pull request as ready for review October 21, 2026 20:15
@cvas22
cvas22 requested a review from a team as a code owner October 21, 2026 20:15
@cvas22
cvas22 requested a review from alanprot October 21, 2026 20:15
Comment thread CHANGELOG.md Outdated
@cvas22
cvas22 force-pushed the fix/7885-orphaned-deletion-marks branch from e000cf4 to 0355207 Compare October 22, 2026 02:21
When a block no longer exists but its deletion mark is still in the
tenant's markers/ location, the bucket index updater skipped the mark
at every run and never removed it, so the tenant location stayed in the
bucket forever. Delete such a mark when the block location is empty.

Fixes cortexproject#7885

Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
@cvas22
cvas22 force-pushed the fix/7885-orphaned-deletion-marks branch from 0355207 to 8f727a8 Compare October 22, 2026 19:47
@cvas22

cvas22 commented Oct 22, 2026

Copy link
Copy Markdown
Author

Rebased onto master to resolve the CHANGELOG.md conflict with #7888 (both entries kept; the code is unchanged). I also fixed the issue link in the description: this fixes #7885.

The earlier test (amd64) failure was TestCPUCheckedBeforeHeap in pkg/util/queryeviction, which this PR doesn't touch; it looks like the flake #7817 addresses. bucketindex and the compactor TestBlocksCleaner tests pass on Linux after the rebase.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compactor leaves orphaned global block deletion markers, keeping empty tenant folders forever

2 participants