Personal deploy slow vs replicate's hosted API

[WesleyKapow]

I'd like to get this setup as my own API so that I can make changes / extend it.

I deployed the dev.yaml config to replicate using an A100 GPU. It works but predictions take about 18s (after booting and being warm, this is 18s active time) where the one at https://replicate.com/black-forest-labs/flux-dev takes about 3s. This won't do :(.

I'm deploying from a windows 11 machine using cog.

Any idea as to what could be going wrong? Even without "go_fast" checked on the hosted API, it only takes 6.4s. Is the hosted one on a MUCH faster GPU?

Ideally I'd be able to get this one up and running on replicate and going just as fast as the hosted one and then I can extend it.

Thanks for any help here!

[josephrocca]

I was wondering the same thing, and after some hunting found there's a table in the repo upon which this repo is based, which shows that H100s GPUs are way faster than 4090 and previous generation GPUs: https://github.com/aredden/flux-fp8-api?tab=readme-ov-file#speed-comparison

For 1024x1024, it's 3.5 it/s for 4090 vs 11.5 it/s for H100. I'm guessing H100 has native support for some FP8 stuff.

I recommend checking out SVDQuant btw - seems like best perf option right now for non-H100 owners based on my testing:

• https://github.com/mit-han-lab/ComfyUI-nunchaku/
• https://github.com/mit-han-lab/nunchaku
• https://github.com/mit-han-lab/deepcompressor/tree/main/examples/diffusion