Skip to content

Promote the parallelism of the stream distribution #1

Description

@JerryLX

Is your feature request related to a problem? Please describe.
Currently, each worker corresponds to a thread for pushing stream blocks.
If there are few requesters, such a method achieves very low bandwidth utilization.
E.g., only one thread is initiated to send stream blocks if there is only one stream requester.
A better choice is to let the number of threads be requester-irrelevant.

Describe the solution you'd like
Similar to the bitswap protocol, gather all sending tasks into a task pool and initiate N task workers keeping pop tasks from the task pool.

Describe TODOs you've considered

  1. Add a task queue in "stream" lib
  2. the "stream_worker" now should do noting but adding tasks in the task queue
  3. add the block_worker, N threads will be initiate when starting stream_service
  4. adding a task in the task queue will notify the block_worker

Additional context
If we parallelize the stream block sending, something bad will happen.
Root blocks and the "ENDMARK" could be received before other data blocks.
More procedures have to be modified.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions