BF: restrict value of get_fdmax

[yarikoptic]

In my case I kept finding celery running at 100% and doing nothing. py-spy pointed to the close_open_fds and then ulimit inside the container showed gory detail of

❯ docker run -it --rm --entrypoint bash dandiarchive/dandiarchive-api -c "ulimit -n"
1073741816

situation is not unique to me. See more at

• test_publish_and_manipulate and test_get_dandiset_published fail locally dandi/dandi-cli#1488

I verified that with this fix my celery container gets unstuck and proceeds to report useful errors ;)

[yarikoptic]

hm, why pre-commit.ci is even configured if there is no .pre-commit-config.yaml?

[auvipy]

lets ignore the pre commit. can you elaborate more on the change please? also should we also consider adding some tests to verify the proposed changes?

[yarikoptic]

I would be happy to elaborate! ATM I can only reiterate what tried to describe in original description -- on some systems ulimit would return HUGE number for maximal number of open descriptiors, which would be infeasible to loop through. So, billiard should not try to loop through all the possible billion of them.

[auvipy]

May be we can add some unit tests for the suggested changes as well

[yarikoptic]

sorry -- too busy ATM, feel welcome to contribute some by directly pushing to my branch for this PR, I did

[Copilot]

Pull Request Overview

The PR adds logic to cap and warn about excessively high file descriptor limits returned by get_fdmax, preventing performance issues when iterating open descriptors.

• Capture os.sysconf('SC_OPEN_MAX') into fdmax and handle errors uniformly
• Introduce a threshold (100k) to cap fdmax to a sensible default (either the passed default or 10 000) and emit a warning
• Import warnings and emit a deprecation-style warning instead of returning an oversized limit

Comments suppressed due to low confidence (1)

billiard/compat.py:118

• Consider adding unit tests for the new high-value cap branch to verify that large fdmax values produce the expected warning and capped return value.

    if fdmax >= 1e5:

[Copilot]

The magic numbers 1e5 and 10000 should be extracted to named constants for clarity and easier future tuning.

Suggested change

	if fdmax >= 1e5:
	# limiting value is ad-hoc and already more than sensible
	if default:
	fdmax_limited = default
	msg = "the default"
	else:
	fdmax_limited = 10000
	if fdmax >= FD_MAX_LIMIT_THRESHOLD:
	# limiting value is ad-hoc and already more than sensible
	if default:
	fdmax_limited = default
	msg = "the default"
	else:
	fdmax_limited = FD_MAX_DEFAULT_LIMIT