Skip to content

Conversation

@quasi
Copy link
Owner

@quasi quasi commented Jun 9, 2025

This commit introduces a distributed pool type (distributed-pool) to the pooler library, utilizing consistent hashing to distribute items across multiple underlying pool instances (nodes).

Key features and changes:

  1. distributed-pool Structure:

    • Manages a collection of named nodes, where each node is a standard pool instance.
    • Uses a consistent hash ring to map keys to specific nodes.
    • Includes configurable replica counts for nodes on the hash ring to improve key distribution.
  2. Consistent Hashing Implementation:

    • hash-ring-node structure to represent nodes on the ring.
    • Functions to add (add-node-to-ring), remove (remove-node-from-ring), and retrieve (get-node-id-for-key-from-ring) nodes based on hashed keys.
    • Uses sxhash for string hashing.
  3. Core Distributed Pool API:

    • make-distributed-pool: Constructor to create a distributed pool with specified node configurations (name, item-maker, capacity, etc.).
    • fetch-from-distributed-pool: Fetches an item from the appropriate node based on a provided key.
    • return-to-distributed-pool: Returns an item to the appropriate node based on a key.
  4. Node Management:

    • add-node-to-distributed-pool: Dynamically adds a new node (pool instance) to an existing distributed pool.
    • remove-node-from-distributed-pool: Removes a node from a distributed pool. (Note: Item draining/migration is not currently implemented for removed nodes).
  5. Utility and Export:

    • get-node-for-key: Retrieves the actual pool object for a given key (used internally and available for advanced use).
    • All new user-facing functions and structures are exported in package.lisp.
  6. Testing:

    • A new test suite (tests/test-pooler.lisp) has been added.
    • Tests cover basic distributed pool operations, node addition/removal, key distribution characteristics, and error handling.
    • pooler.asd has been updated to define and run these tests via (asdf:test-system :pooler/tests).

This implementation provides a foundational distributed pooling mechanism. Future enhancements could include more sophisticated item draining on node removal, dynamic adjustment of replicas, or different hashing strategies.

This commit introduces a distributed pool type (`distributed-pool`) to the pooler library, utilizing consistent hashing to distribute items across multiple underlying pool instances (nodes).

Key features and changes:

1.  **`distributed-pool` Structure**:
    *   Manages a collection of named nodes, where each node is a standard `pool` instance.
    *   Uses a consistent hash ring to map keys to specific nodes.
    *   Includes configurable replica counts for nodes on the hash ring to improve key distribution.

2.  **Consistent Hashing Implementation**:
    *   `hash-ring-node` structure to represent nodes on the ring.
    *   Functions to add (`add-node-to-ring`), remove (`remove-node-from-ring`), and retrieve (`get-node-id-for-key-from-ring`) nodes based on hashed keys.
    *   Uses `sxhash` for string hashing.

3.  **Core Distributed Pool API**:
    *   `make-distributed-pool`: Constructor to create a distributed pool with specified node configurations (name, item-maker, capacity, etc.).
    *   `fetch-from-distributed-pool`: Fetches an item from the appropriate node based on a provided key.
    *   `return-to-distributed-pool`: Returns an item to the appropriate node based on a key.

4.  **Node Management**:
    *   `add-node-to-distributed-pool`: Dynamically adds a new node (pool instance) to an existing distributed pool.
    *   `remove-node-from-distributed-pool`: Removes a node from a distributed pool. (Note: Item draining/migration is not currently implemented for removed nodes).

5.  **Utility and Export**:
    *   `get-node-for-key`: Retrieves the actual pool object for a given key (used internally and available for advanced use).
    *   All new user-facing functions and structures are exported in `package.lisp`.

6.  **Testing**:
    *   A new test suite (`tests/test-pooler.lisp`) has been added.
    *   Tests cover basic distributed pool operations, node addition/removal, key distribution characteristics, and error handling.
    *   `pooler.asd` has been updated to define and run these tests via `(asdf:test-system :pooler/tests)`.

This implementation provides a foundational distributed pooling mechanism. Future enhancements could include more sophisticated item draining on node removal, dynamic adjustment of replicas, or different hashing strategies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants