feat: Implement distributed pool with consistent hashing #4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a distributed pool type (
distributed-pool) to the pooler library, utilizing consistent hashing to distribute items across multiple underlying pool instances (nodes).Key features and changes:
distributed-poolStructure:poolinstance.Consistent Hashing Implementation:
hash-ring-nodestructure to represent nodes on the ring.add-node-to-ring), remove (remove-node-from-ring), and retrieve (get-node-id-for-key-from-ring) nodes based on hashed keys.sxhashfor string hashing.Core Distributed Pool API:
make-distributed-pool: Constructor to create a distributed pool with specified node configurations (name, item-maker, capacity, etc.).fetch-from-distributed-pool: Fetches an item from the appropriate node based on a provided key.return-to-distributed-pool: Returns an item to the appropriate node based on a key.Node Management:
add-node-to-distributed-pool: Dynamically adds a new node (pool instance) to an existing distributed pool.remove-node-from-distributed-pool: Removes a node from a distributed pool. (Note: Item draining/migration is not currently implemented for removed nodes).Utility and Export:
get-node-for-key: Retrieves the actual pool object for a given key (used internally and available for advanced use).package.lisp.Testing:
tests/test-pooler.lisp) has been added.pooler.asdhas been updated to define and run these tests via(asdf:test-system :pooler/tests).This implementation provides a foundational distributed pooling mechanism. Future enhancements could include more sophisticated item draining on node removal, dynamic adjustment of replicas, or different hashing strategies.