Is your feature request related to a problem? Please describe.
Currently, each worker corresponds to a thread for pushing stream blocks.
If there are few requesters, such a method achieves very low bandwidth utilization.
E.g., only one thread is initiated to send stream blocks if there is only one stream requester.
A better choice is to let the number of threads be requester-irrelevant.
Describe the solution you'd like
Similar to the bitswap protocol, gather all sending tasks into a task pool and initiate N task workers keeping pop tasks from the task pool.
Describe TODOs you've considered
- Add a task queue in "stream" lib
- the "stream_worker" now should do noting but adding tasks in the task queue
- add the block_worker, N threads will be initiate when starting stream_service
- adding a task in the task queue will notify the block_worker
Additional context
If we parallelize the stream block sending, something bad will happen.
Root blocks and the "ENDMARK" could be received before other data blocks.
More procedures have to be modified.
Is your feature request related to a problem? Please describe.
Currently, each worker corresponds to a thread for pushing stream blocks.
If there are few requesters, such a method achieves very low bandwidth utilization.
E.g., only one thread is initiated to send stream blocks if there is only one stream requester.
A better choice is to let the number of threads be requester-irrelevant.
Describe the solution you'd like
Similar to the bitswap protocol, gather all sending tasks into a task pool and initiate N task workers keeping pop tasks from the task pool.
Describe TODOs you've considered
Additional context
If we parallelize the stream block sending, something bad will happen.
Root blocks and the "ENDMARK" could be received before other data blocks.
More procedures have to be modified.