Skip to content

Implement hybrid mode for Store Gateway - #7689

Open
SungJin1212 wants to merge 3 commits into
cortexproject:masterfrom
SungJin1212:hybrid-SG
Open

SungJin1212 wants to merge 3 commits into
cortexproject:masterfrom
SungJin1212:hybrid-SG

Conversation

@SungJin1212

@SungJin1212 SungJin1212 commented Jul 14, 2026 •

Copy link
Copy Markdown
Member

What this PR does:

Problem Statement

In the current Parquet mode, the store-gateway only serves blocks that have already been converted to Parquet. Any block that hasn't been converted yet is not queryable at all, since the Parquet store has no way to read TSDB blocks.

In practice, this forces operators to either stay on tsdb mode or wait for full conversion of all historical blocks before flipping the switch. Even then, the gap doesn't fully go away: new blocks are produced continuously (from Compactor) and take some time to be converted.

Solutions

This PR adds a hybrid mode to the Parquet bucket store. The store-gateway runs both a Parquet store and a TSDB sub-store and routes each requested block to one of them, so every block is queryable regardless of its conversion state.

  • Sync: the TSDB sub-store syncs only blocks that are not converted yet. IgnoreParquetBlocksFilter excludes converted blocks using the bucket index.
  • Routing: the same filter records the Parquet block IDs from the bucket index snapshot it used. A block is routed to the Parquet store iff it is recorded there, so excluding a block from the TSDB sub-store and routing it to Parquet always use the same snapshot. There is no gap during the TSDB -> Parquet handover. Before a tenant's first sync, all blocks are routed to the TSDB sub-store, which matches the tsdb mode behavior.
  • Merging: when a request spans both stores, Series queries both concurrently and stream-merges their label-sorted results without buffering either side. LabelNames/LabelValues results and the response hints (QueriedBlocks, QueryStats) are merged as well. If one sub-store fails, its own error is returned so the querier can still classify it (retry, limit, access denied).

Configuration

  • Hybrid mode is the default for -blocks-storage.bucket-store.bucket-store-type=parquet and requires the bucket index. The config validation fails if the bucket index is disabled.
  • New experimental flag -blocks-storage.bucket-store.parquet-fallback-disabled (default false). Setting it to true keeps the previous Parquet-only behavior: no TSDB sub-store, no local state, and the bucket index is not required.

Behavior change

With the default hybrid mode, the Parquet bucket store changes from stateless to semi-stateful: it keeps index-headers on local disk for blocks that are not converted yet. A block that is later converted has its index-header downloaded while the conversion is pending and removed once the store-gateway syncs the converted state.

New metrics

  • cortex_hybrid_bucket_stores_blocks_routed_total{store}: requested blocks routed to each sub-store.
  • cortex_hybrid_bucket_stores_operations_total{store, method}: operations served by parquet, tsdb or mixed.
  • cortex_blocks_meta_synced{state="parquet-converted"}: blocks excluded from the TSDB sub-store because they are converted.

Which issue(s) this PR fixes:
Fixes #7140

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

@SungJin1212
SungJin1212 marked this pull request as draft July 14, 2026 07:34
@SungJin1212
SungJin1212 force-pushed the hybrid-SG branch 4 times, most recently from ce7c27f to ccb6de4 Compare July 21, 2026 02:44
@SungJin1212
SungJin1212 force-pushed the hybrid-SG branch 3 times, most recently from d1d08b0 to 4f3054a Compare July 28, 2026 06:38
@SungJin1212
SungJin1212 marked this pull request as ready for review August 11, 2026 05:53
@CharlieTLe
CharlieTLe requested review from a team and danielblando and removed request for a team August 21, 2026 18:26
@SungJin1212
SungJin1212 requested a review from a team as a code owner October 23, 2026 08:10
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212

Copy link
Copy Markdown
Member Author

#7892 should be merged to fix TestParquetStoreGateway_HybridModeFuzz .

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Parquet] Store Gateway Hybrid Mode

1 participant