Skip to content

Keep buffered rows when a database dump chunk is flushed - #1482

Open
RaphaelFakhri wants to merge 1 commit into
planetscale:mainfrom
RaphaelFakhri:fix-dump-rows-at-chunk-boundary
Open

RaphaelFakhri wants to merge 1 commit into
planetscale:mainfrom
RaphaelFakhri:fix-dump-rows-at-chunk-boundary

Conversation

@RaphaelFakhri

Copy link
Copy Markdown

Summary

pscale database dump with SQL output can silently drop the last rows of a table.

The SQL writer groups rows into an INSERT statement every StmtSize bytes and writes a chunk file every ChunksizeInMB. Flush wrote only the completed INSERT statements and reset the chunk byte counter, leaving the remaining buffered rows for the next file. When the table ended right after that flush, Close saw a chunk byte count of 0 and returned without writing those rows. With the default settings, up to about 1 MB of rows at the end of a table could be missing from the dump without an error.

Changes

  • Flush now turns the buffered rows into a final INSERT statement before it writes the chunk file, so each file is complete and no rows are carried over.
  • The INSERT construction moves into a small appendInsert helper that WriteRow and Flush share; Close no longer needs its own copy.
  • Adds TestDumperKeepsRowsAtChunkBoundary, which dumps a table where the last row reaches the chunk size without completing an INSERT, and checks that every row is in the output files.

Testing

go test ./internal/dumper/ -run TestDumperKeepsRowsAtChunkBoundary -count=1
  • Without the fix: fails with got: int(1040), want: int(1043).
  • With the fix: passes.

go test ./internal/dumper/ ./internal/cmd/database/ passes, and golangci-lint run ./internal/dumper/... reports 0 issues.

Fixes #1481

@RaphaelFakhri
RaphaelFakhri requested a review from a team as a code owner October 22, 2026 12:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pscale database dump drops rows at a chunk boundary

1 participant