[Q&A] Ratchet vs Candle

[DavidGOrtega]

@FL33TW00D I know ratchet prior to be a HF project (congrats to you for joining 🤗 !!!) however the question that I have is how this project relates to Candle. To me, they have pretty common objectives.

[FL33TW00D]

Hi @DavidGOrtega,
There are a few key differences between Candle & Ratchet:

1. We are focused entirely on inference with WebGPU, and as they say "through specialisation comes efficiency".
2. Huge focus on being cross platform. If we don't run with GPU acceleration out of the box on Windows, MacOS and Linux, that's a bug. I think right now we are one of the only frameworks with DirectX support, for speedy inference on Windows.
3. We don't want to just run the model for you. We want to build a toolkit that makes building web applications with AI integrated really easy. This means managing model caching, auto fetching from the hub, versioning, providing tooling for audio & video etc. etc.
4. We have 0 support for training, unlike Candle, which allows us to make some optimisations that aren't currently done in Candle.

More than the above, we wanted to explore a different part of the design space, which can be easier separately. We are hoping one day to merge the best ideas from Candle & Ratchet 🤝. There are tons of places where Candle vastly surpasses us.

[EricLBuehler]

That is good to know! Do you have any benchmarks (even qualitative) of how lazy evaluation improved Ratchet's performance, or was Ratchet designed to be lazily evaluated from the ground up?

[FL33TW00D]

@EricLBuehler Lazy from day 0! You can disable the inplace operations with a flag to see the impact they have (it's substantial).

[EricLBuehler]

Great! Do you have any numbers (1x, 1.5x)?

[FL33TW00D]

@EricLBuehler Not really unfortunately, I don't have many non-inplace kernels, as I don't really need them.
I would expect its ~20%.

[EricLBuehler]

Nice, excellent work here!

[DavidGOrtega]

Huge focus on being cross platform. If we don't run with GPU acceleration out of the box on Windows, MacOS and Linux, that's a bug. I think right now we are one of the only frameworks with DirectX support, for speedy inference on Windows.

I also think that going with wgpu is the way to go. However thats seems to be also Candle's objetive .
Could be Ratchet the missing backend?

[FL33TW00D]

I think yes one day the work from Ratchet could end up as a backend for Candle!