Dragonfly v2.6 Roadmap

[gaius-qi]

Dragonfly

Core Components

Manager

• Enhance service performance and resource utilization while reducing CPU/Memory overhead.
• Enhance user experience and UI design in the Manager Console.
• Support optionally removing the Manager dependency, reducing the minimum cluster deployment unit to Scheduler and Client.

Scheduler

• Enhance service performance and resource utilization while reducing CPU/Memory overhead.
• Optimize the scheduling algorithm to improve bandwidth utilization in the P2P network.

Client

• Enhance service performance and resource utilization while reducing CPU/Memory overhead.
• Implement a bandwidth-aware negotiation protocol to distribute requests across multiple parent nodes,
  preventing single-parent bottlenecks.
• Dfcache/Dfstore support to import persistent cache task to Dfdaemon in Node by UDS.
• Implement RDMA-based distribution of the files.
• Support cache task memory-level download tasks.
• Support reflink to avoid unnecessary data copying.

AI Model/Dataset Distribution

• Writes use Direct IO, reads use Buffered IO.
• Implement Python SDK to provide data distribution for AI Infrastructure.

AI Agent

• Ensure data reliability when asynchronously writing to object storage.
• Implement Python SDK to support snapshotter for use in AI agent.
• Explore integrating the Agent Sandbox, Gymnasium, etc.

Others

Observability

• Improve and refine the monitoring metrics system.
• Optimize the alerting mechanism and enhance issue diagnosis capabilities.

Security

• Implement encrypted data storage.

Testing

• Add more E2E tests and unit tests.

Skills / Agent Capabilities

• Add a Dragonfly skill to enable troubleshooting and diagnosis capabilities.

Documentation

• Enhance the landing page UI.
• Add more documentation on system interactions and implementation details.