GH-50482: [Python] Wait for CPU tasks before interpreter finalization

[mikamikasuki]

Rationale for this change

Fixes #50482. A Python-backed dataset scan can return ArrowInvalid while CPU-pool workers still retain Python-backed resources. During interpreter finalization, the main thread waits for pool shutdown as a worker attempts to attach to Python, so the process can hang or abort.

What changes are included in this PR?

Register an atexit callback that releases the GIL while waiting for the CPU thread pool to become idle before interpreter finalization. Add a subprocess regression test for an FSSpec-backed dataset containing one malformed Parquet fragment.

Are these changes tested?

• Built Arrow and editable PyArrow from this branch on macOS.
• pyarrow/tests/test_misc.py: 123 passed, 1 skipped on CPython 3.12.15 and 3.14.8.
• The targeted exit regression passed 12/12 runs on each Python version.
• On PyArrow 26.0.0, the baseline timed out in 8/8 runs on CPython 3.12.15 and 7/8 runs on 3.14.8.

Are there any user-facing changes?

The malformed dataset continues to raise ArrowInvalid; after the error, interpreter shutdown now waits for Python-backed cleanup to finish.

This PR contains a "Critical Fix." It prevents the reported process-exit hang or abort when worker cleanup attaches to Python during interpreter finalization.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

AI assistance was used to trace the thread-pool shutdown path, implement the wait callback and regression test, and draft this description. The generated changes were reviewed and built on CPython 3.12.15 and 3.14.8.

• GitHub Issue: [Python] Interpreter hangs at exit after pq.read_table(python_file_object) - thread pool teardown deadlock #50482

[github-actions]

⚠️ GitHub issue #50482 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #50482 has no components, please add labels for components.