[jkaczman]

If we have a query we're re-writing for INSERT split that has multiple tuples that resolve to the same shard, this now groups those into a singular query, as opposed to the previous code always splitting each individual tuple into its own statement. This should produce massive speedup for large splits.

Motivated by subqueries, where we'll have cases like INSERT INTO .. SELECT FROM generate_series(..), which will be passed through here and involve massive amount of data.

Future work here will be figuring out how to handle those massive cases further, e.g., we'll need to impose a per-query limit of how much we batch depending on Postgres's limits (w.r.t bind parameters and query sizes). Also, need to figure out a good way to pass the evaluated subquery data rows and row description into the Query Engine without re-writing the top-level statement when we have an INSERT split, so that we don't have to put together unnecessary intermediate string(s), which will just be re-written again to be split apart by INSERT split.

Fixes #1818.

[codecov]

Codecov Report

❌ Patch coverage is 97.60479% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...frontend/router/parser/rewrite/statement/insert.rs	97.22%	3 Missing ⚠️
...uter/parser/rewrite/statement/insert_split_plan.rs	97.43%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Bencher Report

Project	PgDog
Branch	jk-insert-split-by-shard
Testbed	Intel v1

⚠️ WARNING: Truncated view!

The full continuous benchmarking report exceeds the maximum length allowed on this platform.

🐰 View full continuous benchmarking report in Bencher

[github-actions]

Bencher Report

Project	PgDog
Branch	jk-insert-split-by-shard
Testbed	Intel v1

Click to view all benchmark results

Benchmark	Throughput	Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%)	Lower Boundary operations / second (ops/s) x 1e3 (Limit %)	cpu-per-txn	Benchmark Result Measure (units) (Result Δ%)	Upper Boundary Measure (units) (Limit %)
pooler-v2	📈 view plot 🚷 view threshold	29.21 ops/s x 1e3 (-0.36%) Baseline: 29.31 ops/s x 1e3	29.02 ops/s x 1e3 (99.36%)	📈 view plot 🚷 view threshold	34.25 units (+0.36%) Baseline: 34.13 units	34.47 units (99.37%)

🐰 View full continuous benchmarking report in Bencher