Use os.closerange instead of counting to 1 billion#425
Open
phihag wants to merge 1 commit intocelery:mainfrom
Open
Use os.closerange instead of counting to 1 billion#425phihag wants to merge 1 commit intocelery:mainfrom
phihag wants to merge 1 commit intocelery:mainfrom
Conversation
In `billiard.compat.close_open_fds`, after checking whether os.closerange exists, if it exists we did _not_ use it, which makes no sense at all.
Instead, the implementation called `os.close`, `os.sysconf("SC_OPENMAX") - len(keep)` times.
On most systems, SC_OPENMAX is something like 1024, 4096, or 65536.
In the latter case, calling it will already take 30ms on my system, almost noticable for humans.
But docker (and some systems) uses a really high value:
```
docker run --rm python python -c 'import os;print(os.sysconf("SC_OPEN_MAX"))'
1073741816
```
In other words, when running in docker we were counting to 1 billlion, at 2 million per second.
The single function call `close_open_fd([0,1,2])` takes almost 10 minutes there!
The fix is trivial: When `os.closerange` is available, use the existing implementation. It's right there!
An alternative approach would be to just list all open handles with `psutil.Process().open_files()` or so, and then just close those few.
auvipy
requested changes
May 15, 2025
Member
auvipy
left a comment
There was a problem hiding this comment.
can you please add unit tests to verify this? and there is no regression?
There was a problem hiding this comment.
Pull Request Overview
This PR simplifies close_open_fds by removing the manual per-FD fallback loop and always using the existing closerange helper, improving performance especially on systems with very high SC_OPEN_MAX.
- Always invoke
closerangeto close descriptor ranges instead of iterating each FD - Removed the
elseblock that manually closed each FD in reverse - No functional changes to
get_errno
Comments suppressed due to low confidence (4)
billiard/compat.py:130
- The custom
closerangeimplementation still iterates over each file descriptor. To leverage the optimized system call when available, consider aliasingcloserange = os.closerangeifos.closerangeexists or callingos.closerangedirectly.
def closerange(fd_low, fd_high): # noqa
billiard/compat.py:141
- [nitpick] The variable names
kLandkHare not descriptive. Consider renaming them to something likelower_boundsandupper_bounds(orstartsandends) to clarify their purpose.
for low, high in zip_longest(kL, kH):
billiard/compat.py:134
- [nitpick] Add a docstring for
close_open_fdsto describe its behavior, parameters (keeplist), and side effects, improving maintainability.
def close_open_fds(keep=None):
billiard/compat.py:134
- Include unit tests to verify that
close_open_fdscorrectly closes ranges viacloserangefor very large FD limits and respects thekeeplist.
def close_open_fds(keep=None):
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In
billiard.compat.close_open_fds, after checking whether os.closerange exists, if it exists we did not use it, which makes no sense at all.Instead, the implementation called
os.close,os.sysconf("SC_OPENMAX") - len(keep)times.On most systems, SC_OPENMAX is something like 1024, 4096, or 65536.
In the latter case, calling it will already take 30ms on my system, almost noticable for humans.
But docker (and some systems) uses a really high value:
In other words, when running in docker we were counting to 1 billlion, at 2 million per second.
The single function call
close_open_fd([0,1,2])takes almost 10 minutes there!The fix is trivial: When
os.closerangeis available, use the existing implementation. It's right there!An alternative approach would be to just list all open handles with
psutil.Process().open_files()or so, and then just close those few.