Resolve Backpressure Issues about the Task Channel in Node #515
Description
What feature is it?
Implement an adaptive "Task Buffer" with a go slice that efficiently manages task flow between the producer(data-source) and the consumer(worker). The producer adds tasks to the buffer and pauses when the buffer is full. The consumer then takes and processes tasks from the buffer.
What problem does this feature solve?
This mechanism ensures that all collected data is processed successfully and efficiently under backpressure conditions. Currently, the task channel is unbuffered, which can cause delays and blocking latency when too many tasks are sent to the channel in a short period. Implementing a sliding window buffer makes data processing more efficient.
Additional description
No response
This is not a duplicated feature request or new worker proposal
- I have searched existing issues to ensure this feature has not already been requested and this is not a New Worker.