Stable Diffusion XL much slower with candle than with diffusers

I already tried to change the float type to f16 for the decover, the change is barely noticeable.

As suggested, I’ve tried patching the scheduler, using the same configuration as found in the scheduler.json in the repository of the model, but that changes nothing: the speed is exactly the same as before, and I still get weirdly off looking images that I don’t get with diffusers.

I think the next step will be to try to use the diffusers-rs library, which seems to be backed by Torch rather than candle.

Incidentally, I tried to use Torch compilation with the diffusers library and that resulted in a huge speedup. With it, the diffusion steps can reach the speed of 4 steps per second.