Repository navigation
[Python][C++] Deadlock in ensure_s3_finalized atexit handler when pyarrow.fs.S3FileSystem reference outlives interpreter shutdown (pyarrow 24.x regression) #50188
Description
Describe the bug, including details regarding any error messages, version, and platform.
In pyarrow==24.0.0, a process that creates a pyarrow.fs.S3FileSystem and keeps a reference to it until interpreter shutdown will deadlock during Py_FinalizeEx. The hang is in the ensure_s3_finalized atexit handler registered by pyarrow/fs.py, inside Aws::Crt::Io::ClientBootstrap::~ClientBootstrap(). The same code exits cleanly on pyarrow==23.0.0.
This is the AWS-CRT "blocking-shutdown" deadlock that aws-sdk-cpp documents in aws/aws-sdk-cpp#2769 — but it is reachable from pure-Python pyarrow without any explicit S3 I/O, just from holding a S3FileSystem Python reference past interpreter shutdown. Arrow's atexit-driven finalize ordering plus the AWS C/C++ stack version bumps in 24.0.0 turn this from a quiet teardown into an indefinite hang.
Reproducer
# bug.py
import pyarrow.fs
s3 = pyarrow.fs.S3FileSystem()
print("S3FileSystem created, exiting...")$ python bug.py
S3FileSystem created, exiting...
# process never exits; SIGTERM required
No S3 request is issued — just constructing S3FileSystem() and holding the reference is enough.
Diagnostic: del s3 fixes it; pyarrow.fs.finalize_s3() does not
# fixed.py — clean exit
import pyarrow.fs
s3 = pyarrow.fs.S3FileSystem()
print("S3FileSystem created, exiting...")
del s3# still_hangs.py — same deadlock, just earlier in the script
import pyarrow.fs
s3 = pyarrow.fs.S3FileSystem()
pyarrow.fs.finalize_s3() # hangs here instead of at atexitThat asymmetry pins the cause to "live S3Client reference at the moment Aws::ShutdownAPI runs," not to a missed wakeup or a broken finalizer.
gdb backtrace — main thread (the deadlock victim)
Thread 1 (Thread 0x7ff7151f5740 (LWP 3685) "python"):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007ff70fe02671 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (...)
from /lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007ff7114d8f01 in Aws::Crt::Io::ClientBootstrap::~ClientBootstrap()
from pyarrow/libarrow.so.2400
#3 0x00007ff711475b91 in Aws::SetDefaultClientBootstrap(...)
from pyarrow/libarrow.so.2400
#4 0x00007ff711475bc9 in Aws::CleanupCrt() from pyarrow/libarrow.so.2400
#5 0x00007ff7114735a5 in Aws::ShutdownAPI(...) from pyarrow/libarrow.so.2400
#6 0x00007ff710a04a28 in arrow::fs::EnsureS3Finalized()
from pyarrow/libarrow.so.2400
#7 0x00007ff6fe6693c0 in __pyx_pw_7pyarrow_5_s3fs_7ensure_s3_finalized(...)
from pyarrow/_s3fs.cpython-312-x86_64-linux-gnu.so
#8 0x000055bdcca34c2c in atexit_callfuncs at Modules/atexitmodule.c:137
#9 0x000055bdcca22213 in _PyAtExit_Call at Modules/atexitmodule.c:157
#10 Py_FinalizeEx () at Python/pylifecycle.c:1927
#11 0x000055bdcca30920 in Py_RunMain () at Modules/main.c:716
#12 0x000055bdcc9ea477 in Py_BytesMain (...) at Modules/main.c:768
gdb backtrace — the AWS CRT event-loop thread
Thread 66 (Thread 0x7ff661ff0640 (LWP 3751) "AwsEventLoop1"):
#0 0x00007ff71531feae in epoll_wait (epfd=4, ..., timeout=100000)
at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007ff71153cf1a in aws_event_loop_thread () from pyarrow/libarrow.so.2400
#2 0x00007ff7115fe639 in thread_fn () from pyarrow/libarrow.so.2400
#3 0x00007ff71528eac3 in start_thread (...) at pthread_create.c:442
#4 0x00007ff71531fa84 in clone () at clone.S:100
The epoll_wait(timeout=100000) is the standard idle poll interval of aws-c-io's Linux epoll event loop (DEFAULT_TIMEOUT = 100 * 1000). It is not the bug signal on its own. The real bug signal is that nothing wrote to the loop's wake pipe/eventfd to ask it to stop — because the underlying aws_client_bootstrap's C-side refcount never reached zero.
Root cause analysis
~ClientBootstrap() in aws-crt-cpp is:
aws_client_bootstrap_release(m_bootstrap);
if (m_enableBlockingShutdown) {
// If your program is stuck here, stop using EnableBlockingShutdown()
m_shutdownFuture.wait();
}(aws-crt-cpp v0.38.0 source/io/Bootstrap.cpp)
Aws::InitAPI() (called by Arrow's InitializeS3) unconditionally calls clientBootstrap->EnableBlockingShutdown() (aws-sdk-cpp 1.11.594 Aws.cpp:90), so the destructor always takes the blocking path.
The libstdc++ frame in our stack is std::future::wait() on m_shutdownFuture — a promise fulfilled by the C-layer's on_shutdown_complete callback, which only fires once the C aws_client_bootstrap reaches refcount zero.
The Python reference s3 keeps pyarrow._s3fs.S3FileSystem alive → keeps the C++ shared_ptr<S3Client> alive → keeps the underlying aws_s3_client alive → which holds a strong reference on aws_client_bootstrap. So when Aws::CleanupCrt() drops Arrow's default shared_ptr<ClientBootstrap> (the only thing the C++ wrapper destructor releases is its own one C-side ref), the C bootstrap still has refcount > 0, on_shutdown_complete never fires, and the main thread futex-waits indefinitely. Meanwhile the event-loop thread sits at the top of its 100 s idle poll, never told to stop.
Why 23.0.0 worked and 24.0.0 doesn't
Arrow's s3fs.cc finalize code is essentially identical between 23.0.0 and 24.0.0; the change is in the bundled AWS C/C++ stack (cpp/thirdparty/versions.txt):
| component | 23.0.0 | 24.0.0 |
|---|---|---|
| aws-crt-cpp | 0.32.8 | 0.38.0 |
| aws-c-io | 0.19.1 | 0.26.3 |
| aws-c-s3 | 0.8.1 | 0.12.0 |
The EnableBlockingShutdown + m_shutdownFuture.wait() pattern itself has not changed across these versions, but the ref-graph / teardown-task scheduling in aws-c-io between 0.19 and 0.26 changed enough that a previously-fast-but-still-incorrect shutdown now reliably deadlocks when any S3Client reference is alive.
Suggested fix directions
- Arrow side (preferred): at the top of
arrow::fs::FinalizeS3(beforeAws::ShutdownAPI), drop Arrow's internalS3Clientcache (S3ClientFinalizer) and ensure noS3FileSystem-ownedshared_ptr<S3Client>survives. Today the finalizer does call intoS3ClientFinalizer::Finalize(), but anyS3FileSysteminstance still reachable from Python keepsS3ClientHolderalive past that point. Either:- hold
S3Clientviaweak_ptrinsideS3FileSystemand re-resolve through the finalizer, or - have the atexit handler eagerly walk known
S3FileSysteminstances (via aweak_ptrregistry) and reset their clients before callingShutdownAPI.
- hold
- Bypass the blocking destructor: consider whether Arrow needs
Aws::InitAPIdefaults at all — buildingSDKOptionswith a customClientBootstrapthat doesn't callEnableBlockingShutdown()would convert this from a hang into a benign leak (matching the warning in aws-crt-cpp's own source). - At minimum, document the footgun. Today there is no docstring warning that holding a
pyarrow.fs.S3FileSystempast interpreter shutdown will hang the process onpyarrow>=24.
Related
- aws/aws-sdk-cpp#2769 —
ShutdownAPIhangs when client outlivesShutdownAPI(closed; identical stack signature, root cause documented) - aws-crt-cpp v0.38.0
~ClientBootstrapsource comment — "If your program is stuck here, stop using EnableBlockingShutdown()" - Arrow PR #38375 (GH-38364) — atexit registration for
ensure_s3_finalized(the path that fires the deadlock)
Component(s)
Python, C++
Note: I captured the gdb dump and verified the reproducer (including the del s3 vs finalize_s3() asymmetry) on a real environment; the write-up and cross-references to aws-sdk-cpp / aws-crt-cpp source were drafted with help from Claude. Happy to provide the full gdb log, or additional traces on request.
Activity
pitrou commentedon Jun 16, 2026
The call to
EnableBlockingShutdownwas added in #34134, do you remember why you did that @westonpace ?pitrou commentedon Jun 16, 2026
@jpatra72 Which Python version are you using and how did you install PyArrow?
pitrou commentedon Jun 16, 2026
(I cannot reproduce locally, I tried with a wheel on Python 3.12 and with a local conda-based build on Python 3.10)
jpatra72 commentedon Jun 16, 2026
@pitrou, this was on py312.
The repro part is a bit tricky in this case. I'm not able to repro this on my local setup as well. The error trace is from one of our Anyscale users who runs ray on a k8s environment (and there are certain parts of their k8s setup that I don't have visibility into). The best info I have here to offer are the trace logs I shared above.
pitrou commentedon Jun 16, 2026
But did the user install PyArrow from a wheel? Something else?
jpatra72 commentedon Jun 16, 2026
it was installed from a wheel -
pyarrow-24.0.0-cp312-cp312-manylinux_2_28_x86_64.whlpitrou commentedon Jun 17, 2026
Thanks, can you ask them to try the development wheel at https://github.com/ursacomputing/crossbow/actions/runs/27629054110/artifacts/7672240236 ?
jpatra72 commentedon Jun 17, 2026
Let me try. Btw, may I know what changes are made here in this develoipment wheel, thanks
pitrou commentedon Jun 17, 2026
It's from this PR: #50193
It removes the
EnableBlockingShutdowncall mentioned in your description above, but also contains all unreleased changes in the current git main.tcsavage commentedon Jun 18, 2026
RE: reproducing the issue @pitrou
I have experienced this issue and in attempting to diagnose the problem myself, have only been able to reproduce it when using AWS STS web identity credentials (in my case running in an EKS cluster (IRSA provides the
AWS_WEB_IDENTITY_TOKEN_FILEenv var).I was NOT able to reproduce using bare access key and secret key, or SSO credentials.
This was all I needed to demonstrate the problem (running in a container with web identity creds provided):
pitrou commentedon Jun 18, 2026
Thanks @tcsavage , would you be able to test with a development wheel of PyArrow? Which Python version are you using?
tcsavage commentedon Jun 18, 2026
@pitrou I was using Python 3.13.14 with PyArrow 24.0.0. I can absolutely test with a development wheel. I'm only set-up for Linux on amd64 or arm64 though.
pitrou commentedon Jun 18, 2026
Ok, you'll be able to download a wheel from the build artifacts from the appropriate CI job once it has finished in #50193 (comment)
tcsavage commentedon Jun 19, 2026
I've tested one of the new wheels (specifically this one) in my harness and it still exhibits the problem (won't let Python exit).
Here's some more information. Hopefully somebody else can replicate this.
Harness
My harness consists of a Docker image I can run inside a pre-existing EKS cluster.
Dockerfile:
This image was built and pushed to an internal registry.
I can launch a pod in the EKS cluster like so:
(The node selector is just to force scheduling on an AMD64 instance because that's what the wheel was built for.)
This pod will print the following, then hang forever until killed:
Traces
Running
pstreeinside the stuck container shows me this:And a GDB trace:
(I think we can ignore the
ModuleNotFoundError. I believe this is coming from GDB itself)tcsavage commentedon Jun 19, 2026
I've attempted to replicate this using web identity tokens outside EKS on my local macOS machine, but have so far been unable to reproduce the issue in that environment.
jpatra72 commentedon Jun 19, 2026
Yup, I see the same end result with this whl.
gdb logs here
pitrou commentedon Jun 22, 2026
@tcsavage Why is
dpkg-preconfigureappearing in your Python process tree?@jpatra72 Your traceback shows
libarrow.so.2400, notlibarrow.so.2500.tcsavage commentedon Jun 24, 2026
@pitrou Fair question. I had
kubectl execed into the container and had to install a package to getpstree. I don't know why those processes would stick around though. The point was really just to show thatpythonwas PID 1 and there's just thejemallocand AWS event loop threads running.exaroj commentedon Jun 24, 2026
I also saw this in production (under pyiceberg on EKS) but cannot reproduce locally.
tcsavage commentedon Jun 24, 2026
@exaroj couple of follow-up questions if you don't mind:
exaroj commentedon Jun 24, 2026
We have spent (wasted) a lot of time trying to debug and reproduce AWS S3 CRT issues in the past. We struggled because the AWS libraries were attempting to determine if they were running on EKS/EC2 instances and tailor behavior accordingly.
jpatra72 commentedon Jun 25, 2026
close it by mistake, @pitrou could you pls reopen
dhirschfeld commentedon Sep 11, 2026
This bug is biting me pretty hard. I want to move to
>=24for new UUID features but this bug is causing our Dagster containers to hang in such a way as they're never cleaned up on k8s (Dagster gets a success signal before the hang happens).The bug only manifests when the containers are run in Dagster Cloud, never locally, making it super hard to track down, and hard to provide any repro - I don't know the exact conditions that trigger the bug for us.
We've now set a cluster-level timeout on the Dagster pods, but have also reverted to 23.
Does anyone have any updated info? A work-around?
ROMNEUVILLE commentedon Sep 14, 2026
This bug is also impacting our workloads. We’re running Kubernetes pods on AWS using IRSA, so pinning to older versions isn’t a viable option for us.
Are there any plans or timelines for a fix? Is there a concrete workaround that doesn’t involve pinning to an older version or adding a hard exit at the end, or an explicit del statement? I
raulcd commentedon Sep 14, 2026
@tcsavage suggests that it only triggers with STS web identity credentials, not with access keys or SSO. Could someone try plain Linux docker (not macOS, the event loop backend differs, and outside K8s) with
AWS_WEB_IDENTITY_TOKEN_FILE,AWS_ROLE_ARNandAWS_REGIONset, pointing at a token file, and see whether it hangs? Even a bogus or expired token may be enough, since the deadlock happens without any S3 request being issued.