Skip to content

Fix concatenating Arrow result chunks with duplicate column names - #994

Open
adityaanikam wants to merge 1 commit into
databricks:mainfrom
adityaanikam:fix-concat-duplicate-columns-989
Open

adityaanikam wants to merge 1 commit into
databricks:mainfrom
adityaanikam:fix-concat-duplicate-columns-989

Conversation

@adityaanikam

Copy link
Copy Markdown

Fixes #989

Since 4.2.6, _concat_arrow_tables always calls pyarrow.concat_tables(..., promote_options="default"). Arrow's schema unification rejects duplicate field names, so any result with two columns of the same name (for example SELECT 1 AS a, 2 AS a) fails on fetch with ArrowInvalid: Can't unify schema with duplicate field names, even when every chunk has the same schema.

The chunks are now concatenated directly when all their schemas are equal. The promote path is only used when the schemas differ, so the nullability case from #418 still works. A result that has duplicate names and differing chunk schemas still fails, as before.

Tests (tests/unit/test_util.py): added a duplicate-column-names test and a nullability-promotion test. I also changed the old-pyarrow fallback test so its second chunk differs in nullability. Otherwise the two chunks would now skip the promote path and the fallback would no longer be exercised. The duplicate-names test fails without the change. pytest tests/unit -m "not realkernel" passes (1264 passed), and black --check is clean. I only ran this on Windows with Python 3.12 and pyarrow 26, not against a real warehouse or with older pyarrow versions.

Added a line to the Unreleased section of CHANGELOG.md.

Signed-off-by: adityaanikam <adityanikam9502@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 20, 2026 18:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Queries returning duplicate column names fail with ArrowInvalid: Can't unify schema with duplicate field names since 4.2.6

2 participants