Skip to content

GH-50142: [C++][Python] Honor Parquet time coercion and format version - #53161

Open
gitedmond wants to merge 2 commits into
apache:mainfrom
gitedmond:gh-50142-time-coercion
Open

gitedmond wants to merge 2 commits into
apache:mainfrom
gitedmond:gh-50142-time-coercion

Conversation

@gitedmond

@gitedmond gitedmond commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Fixes #50142. Writing time64[ns] with Parquet version 1.0 or 2.4 currently emits TIME(NANOS), and explicit coerce_timestamps options leave time-of-day fields unchanged. For example, a value of 1,000 nanoseconds remains TIME(NANOS) instead of becoming 1 microsecond under version 2.4.

Credit to fenfeng9's analysis for separating the compatibility bug from the existing timestamp-only option semantics. This follows the agreed direction to reuse the existing coercion options, rather than adding time-specific options. The footer's reported format_version is not changed.

What changes are included in this PR?

  • Select time annotations and physical widths from the requested coercion unit and format version.
  • Convert values to the selected on-disk units, preserving nulls and rejecting precision loss or integer overflow. Milliseconds use INT32; microseconds and nanoseconds use INT64.
  • Update the existing older-version roundtrip fixture, add focused regressions, and document time-of-day coercion.
  • Propagate conversion errors through both dense and dictionary content-defined chunking write paths so invalid input raises an Arrow exception instead of aborting the process.

Are these changes tested?

  • parquet-arrow-reader-writer-test: 834 passed; two unsupported dictionary cases skipped and one existing large-string test disabled.
  • parquet-chunker-test: 328 passed.
  • parquet-arrow-metadata-test: 49 passed.
  • Python Parquet test_datetime.py, with Parquet and pandas groups enabled, against locally built PyArrow and C++ libraries: 86 passed.
  • Existing Python content-defined chunking tests in test_parquet_writer.py: 2 passed.
  • Fail-before/fix-after: the reviewer's 1001 ns / version 2.4 reproducer and a dictionary-input variant both abort with SIGABRT on the previous PR head (7e8b0297). After the fix, the same subprocess checks catch ArrowInvalid with would lose data and exit normally. The non-chunking control also raises the expected exception.
  • Regression coverage includes versions 1.0/2.4, both chunked write paths, truncation permission without an explicit coercion target, explicit allowed truncation, overflow, null parents, array layouts, and successful conversions spanning multiple content-defined chunks.
  • Clang-format 18.1.8, cpplint 1.6.1, configured Python flake8 checks, and git diff --check pass.

Are there any user-facing changes?

coerce_timestamps and allow_truncated_timestamps now also apply to Arrow time32/time64 fields. Without explicit coercion, nanosecond times are converted to microseconds for versions 1.0/2.4 and preserved for 2.6. Lossy conversion raises an error unless an explicit coercion target and allow_truncated_timestamps=True are both provided. Conversion errors also raise the expected Arrow exception when content-defined chunking is enabled.

Was AI used for this PR?

Codex assisted with the investigation, implementation, tests, documentation and PR description, and reviewed the changes before submission.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50142 has been automatically assigned in GitHub to PR creator.

Comment thread python/pyarrow/tests/parquet/test_datetime.py
@github-actions github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 22, 2026
…king

Return errors from both dense and dictionary chunked writes instead of
aborting on a failed Arrow status. Cover the reported time coercion crash,
truncation and overflow, null handling, and successful multi-chunk writes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python][Parquet] pyarrow.parquet.write_table ignores version and coerce_timestamps arguments for Time64 fields

2 participants