We'd better avoid duplicated copy from JNI/Cython to cpp objects. Current data transporting pipeline: * User writes a object in any schema, invoking collect and passing a bytes format data to `Data Writer` * Data Writer copy and move them to transporting, then send bytes data to other process. <img width="143" alt="image" src="https://user-images.githubusercontent.com/7359537/160365357-cb7a89c9-3d9c-4e81-983b-52f4e5f193f5.png"> There are two copy operations. Actually, we don't need copy them between transfer and data writer, which is why we proposal zero-copy buffer pool. How buffer pool works: * Get a fixed data buffer and wrap them in cython or java ByteBuffer * User function can serilize their data to ByteBuffer directly * Data Writer only capture data's pointer and transfer them in pointer * Release them after upstream consumed and recevied a notification from peer target. <img width="426" alt="image" src="https://user-images.githubusercontent.com/7359537/160366858-3e18cfdb-6d7d-456f-8815-820b6a12d162.png">
We'd better avoid duplicated copy from JNI/Cython to cpp objects.
Current data transporting pipeline:
Data Writer
There are two copy operations.Actually, we don't need copy them between transfer and data writer, which is why we proposal zero-copy buffer pool.
How buffer pool works: