fix(io): pass S3 CA cert to PyArrow

[1fanwang]

Closes #3866

Rationale for this change

The PyArrow FileIO path accepted s3.ssl.ca-cert in catalog properties, but it did not pass the value into PyArrow's S3 client. Users with private S3 endpoints or self signed certificates could still hit curlCode: 60 during PyArrow reads and writes.

This change maps the existing Iceberg property to PyArrow's TLS CA file argument for S3 and OSS filesystems. PyArrow added that argument in 21.0.0, so older PyArrow versions now fail fast only when the CA certificate property is set.

Are these changes tested?

Yes.

Raw logs

Before 545cb66:

$ uv run python -m pytest tests/io/test_pyarrow.py::test_pyarrow_s3_session_properties -v
FAILED [100%]
E Expected: S3FileSystem(..., tls_ca_file_path='/path/to/ca.pem')
E Actual:   S3FileSystem(..., region='us-east-1')
1 failed in 0.63s

After 545cb66:

$ uv run python -m pytest tests/io/test_pyarrow.py::test_pyarrow_s3_session_properties tests/io/test_pyarrow.py::test_pyarrow_s3_ssl_ca_cert_requires_supported_pyarrow_version -v
PASSED [ 50%]
PASSED [100%]
2 passed in 0.40s

Adjacent S3 property checks passed: 4 passed, 241 deselected in 1.54s.

Are there any user-facing changes?

Yes. s3.ssl.ca-cert now takes effect for PyArrow-backed S3 and OSS file operations when PyArrow is 21.0.0 or newer.

AI Disclosure

AI disclosure: GitHub Copilot CLI (Claude Opus 5.5) wrote this change and its tests, and an automated coordinator reviewed the diff and test output before submission.

Areas of uncertainty: none known.

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical runtime failure and an undeclared dependency must be fixed.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Use the existing pyarrow alias instead of undefined pa reference · New

What changed in this PR

Adds support for forwarding s3.ssl.ca-cert to PyArrow S3/OSS filesystems, with PyArrow version validation.

Changes:

• Defines the S3 CA certificate property.
• Passes the CA path to PyArrow.
• Adds propagation and compatibility tests.

Review blockers: the implementation references undefined pyarrow instead of pa, and uses an undeclared runtime packaging dependency.

File	Description
tests/​io/​test_pyarrow.py	Tests CA propagation and version requirements.
pyiceberg/​io/​pyarrow.py	Adds TLS CA configuration and version checks.
pyiceberg/​io/​__init__.py	Defines the CA certificate property.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.