fix(rmw-zenoh-rs): release callback-mutex/user_data locks before calling out - #351
Open
YuanYuYuan wants to merge 2 commits into
Open
YuanYuYuan wants to merge 2 commits into
YuanYuYuan wants to merge 2 commits into
Conversation
Subscription/Service/Client each held callback_holder and/or user_data_holder across a call into the registered notify/response callback -- a self-deadlock if that callback re-enters (on the real rclpy path, the callback wants Python's GIL, and the setter runs on a GIL-holding thread). Collect what the call-out needs, release every lock, then call out. Two locks per entity, not one: an earlier version of this fix only released callback_holder and missed that user_data_holder is a second, independently-contended AB-BA pair with the setter -- caught only by actually running the regression test, not by inspection. Extracts the notify-closure/setter-body logic into pub(crate) functions so it's directly unit-testable, with a real embedded Python interpreter standing in for rclpy's GIL. All three regression tests: real hang without the fix, clean pass (0.02s) with it.
Independent review found a real divergence: the pre-fix subscription setter nested the unread-check/call/reset inside callback_holder's own lock, so a poisoned callback_holder meant nothing ran at all. The first version of this fix computed the pending count independently of that lock, so a poisoned callback_holder now reset unread and fired the callback anyway -- a real, if narrow, behavior change. Restored the nesting (service/client's setters never had this nesting in the original, so they're unaffected). Also strengthens subscription_setter_with_no_pending_messages_does_not_panic to actually assert the callback never fired, not just that it compiled -- the test's own name claimed this but didn't check it. Verified: fmt/clippy clean, all 11 tests pass, 5/5 clean runs (checked for flakiness from the shared test statics running in cargo's default parallel mode).
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Subscription/Service/Clienteach heldcallbackand/orcallback_user_dataacross a call into the registered notify/response callback — a self-deadlock if that callback re-enters (on the realrclpypath, the callback wants Python's GIL, and the setter runs on a GIL-holding thread).The fix
Each notify/setter function now copies out what the call-out needs, releases every lock, and only then calls out — matching the pattern this codebase already uses elsewhere for the same shape:
Extracted the notify-closure and setter-body logic into free functions per entity, so each is directly unit-testable without a live zenoh session or the full RMW FFI chain.
Two locks per entity, not one
The first version of this fix released only the callback lock. That still deadlocked —
callback_user_datais a second, independently-contended lock with the setter, holding it across the call-out reproduces the identical shape. Caught by actually running a regression test with a real embedded Python interpreter standing in for the GIL, not by inspection.Before and after
unread_count(retroactive-notification tracking)Verification
cargo fmt --check/cargo clippy --all-targets -- -D warnings: clean.cargo test -p rmw-zenoh-rs --lib: all tests pass, including a regression test per entity (Subscription/Service/Client) with a real embedded Python interpreter contending for the same locks the realrclpypath would — real hang without this fix, clean pass (0.02s) with it.Breaking changes
None. No public API changes; the fix is purely internal to the notify/setter call-out sites.
Related work
This is one of a small, independent set of public fixes for the same underlying defect shape (a lock held across a call into user/foreign code) across the zenoh/ROS 2 ecosystem: eclipse-zenoh/zenoh#2781 and YuanYuYuan/zenoh#1 (zenoh's own Rust core), and ros2/rmw_zenoh#1061 (the C++ RMW implementation, same shape, independently re-derived). Stacked on this PR: #352, an opt-in lock-tripwire regression guard over the locks this PR cleans up.