Skip to content

Conversation

@d-v-b
Copy link
Contributor

@d-v-b d-v-b commented Oct 24, 2025

Makes the concurrency limit global, rather than per-function.

I'm not emotionally attached to this PR -- it's largely the work of claude code. Lets use this as a starting point for improving on #3526

@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Oct 24, 2025
@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Oct 27, 2025
@dcherian
Copy link
Contributor

Nice, the other one that bugs me when profiling is

return await asyncio.to_thread(

@maxrjones
Copy link
Member

@d-v-b do you have thoughts on how downstream libraries would interact with Zarr's concurrency limit? Would those set there own concurrency approaches (e.g., global or per function) or could there be a way to share the limit with Zarr?

@d-v-b
Copy link
Contributor Author

d-v-b commented Dec 22, 2025

@maxrjones this PR is moving away from a global concurrency limit because it's actually very hard to implement a top-down concurrency control in our codebase. the new direction I'm taking with this PR is for each store to set its own concurrency limit. for stateless stores like local and remote (basically anything except zip), concurrency-sensitive applications should be able to cheaply create store instances with concurrency limits tuned to the needs of the application.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants