Repository navigation
Implement hybrid mode for Store Gateway - #7689
Open
SungJin1212 wants to merge 3 commits into
Open
SungJin1212 wants to merge 3 commits into
SungJin1212 wants to merge 3 commits into
Conversation
SungJin1212
marked this pull request as draft
July 14, 2026 07:34
SungJin1212
force-pushed
the
hybrid-SG
branch
4 times, most recently
from
July 21, 2026 02:44
ce7c27f to
ccb6de4
Compare
SungJin1212
force-pushed
the
hybrid-SG
branch
3 times, most recently
from
July 28, 2026 06:38
d1d08b0 to
4f3054a
Compare
SungJin1212
marked this pull request as ready for review
August 11, 2026 05:53
CharlieTLe
requested review from
a team and
danielblando
and removed request for
a team
August 21, 2026 18:26
SungJin1212
force-pushed
the
hybrid-SG
branch
from
October 23, 2026 08:10
64f6f88 to
0a14e4f
Compare
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
SungJin1212
force-pushed
the
hybrid-SG
branch
from
October 23, 2026 10:17
27c3282 to
aa3dd5c
Compare
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Member
Author
|
#7892 should be merged to fix |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Problem Statement
In the current Parquet mode, the store-gateway only serves blocks that have already been converted to Parquet. Any block that hasn't been converted yet is not queryable at all, since the Parquet store has no way to read TSDB blocks.
In practice, this forces operators to either stay on tsdb mode or wait for full conversion of all historical blocks before flipping the switch. Even then, the gap doesn't fully go away: new blocks are produced continuously (from Compactor) and take some time to be converted.
Solutions
This PR adds a hybrid mode to the Parquet bucket store. The store-gateway runs both a Parquet store and a TSDB sub-store and routes each requested block to one of them, so every block is queryable regardless of its conversion state.
IgnoreParquetBlocksFilterexcludes converted blocks using the bucket index.Seriesqueries both concurrently and stream-merges their label-sorted results without buffering either side.LabelNames/LabelValuesresults and the response hints (QueriedBlocks,QueryStats) are merged as well. If one sub-store fails, its own error is returned so the querier can still classify it (retry, limit, access denied).Configuration
-blocks-storage.bucket-store.bucket-store-type=parquetand requires the bucket index. The config validation fails if the bucket index is disabled.-blocks-storage.bucket-store.parquet-fallback-disabled(defaultfalse). Setting it totruekeeps the previous Parquet-only behavior: no TSDB sub-store, no local state, and the bucket index is not required.Behavior change
With the default hybrid mode, the Parquet bucket store changes from stateless to semi-stateful: it keeps index-headers on local disk for blocks that are not converted yet. A block that is later converted has its index-header downloaded while the conversion is pending and removed once the store-gateway syncs the converted state.
New metrics
cortex_hybrid_bucket_stores_blocks_routed_total{store}: requested blocks routed to each sub-store.cortex_hybrid_bucket_stores_operations_total{store, method}: operations served byparquet,tsdbormixed.cortex_blocks_meta_synced{state="parquet-converted"}: blocks excluded from the TSDB sub-store because they are converted.Which issue(s) this PR fixes:
Fixes #7140
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags