[Bug]: xformers always generate different results, seed is the same!

[mykeehu]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

With the same parameters I get different results for each generation if xformers is enabled. This is a big problem!
If xformers is disabled, the error does not occur.

Steps to reproduce the problem

1. I used these parameters for generation with xformers enabled:

wood table top
Steps: 25, Sampler: Euler a, CFG scale: 7, Seed: 1057871485, Size: 512x512, Model hash: 7460a6fa

2. Generate multiple images, one by one
3. Compare the images.
4. Do the same without xformers

What should have happened?

With the same seed and parameters, get the same result with xformers enabled as without, and never change

Commit where the problem happens

d699720

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

With xformers:
--xformers --ui-config-file 1my/ui-config-my.json --ui-settings-file 1my/config-my.json --autolaunch --gradio-img2img-tool color-sketch


without xformers:
--ui-config-file 1my/ui-config-my.json --ui-settings-file 1my/config-my.json --autolaunch --gradio-img2img-tool color-sketch

Additional information, context and logs

No response

[mykeehu]

Here are some samples, parameters are the same:

I also tested with version 17a2076, but the error persists. The --reinstall-xformers did not help

[ClashSAN]

I recall this always being a thing. Search the other related issues

[ClashSAN]

#3967

[0xdevalias]

A friend mentioned that using xformers could make things non-deterministic, and that there were a lot of references to it on the repo issues here. Wanting to understand a bit more about it, and link a bunch of potentially related issues together, I tried to find as many issues as I could that seemed to be related to xformers and the potential for it to be causing non-deterministic / unstable / inconsistent results:

The following may potentially be related (ordered by issue number):

Issues:

• Xformers optimization causes inconsistent generations with the same parameters #1999
• No token limit for prompts broken? Different results with the same prompts and seed #2005
• The same image can no longer be output under the same conditions. #2197
• On the problem that the same resolution can not be generated stably #2293
• Different results after swapping graphics card #2423
• Different images for same seed with xformers and "Highres. fix" enabled when ancestral samplers are used #2528
• Different outputs since WebUI update #2943
• xformers and euler_a variant results for same generation settings #3054
• [Bug]: The prompts do not produce the same results as before the latest updates on 10/22/22 #3476
• [Bug]: xformers always generate different results, seed is the same! #3967
• [Bug]: In img2img, setting denoising strength to 0 does not return original image #4011

Discussions:

• Different results after swapping graphics card #2418
• --xformers CMD line option causes slightly poorer quality images on GTX 1650 (Turing), possibly others? #2592
• Highres.fix composition #2705
• Exactly same prompt no more the same ? #2849
• xformers option has good and bad aspects #2958
• What has changed recently? Different and Worse Results from Same Settings #3461

I also came across this thread in the xformers repo, which while I can't guarantee is related, am wondering if it might be:

• xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219

And a question I raised on a PR in the diffusers repo:

• Up to 2x speedup on GPUs using memory efficient attention huggingface/diffusers#532 (comment)

  • A friend just mentioned that apparently using xformers tends to lower the quality of generated images, and apparently even causes StableDiffusion to generate different images for the same seed/settings. I haven't looked too deeply into things to try and validate that, but I was wondering if a) that is something that is currently known/documented, and b) if that's likely to be fixed at any point, or if it's just 'the price we pay' for the benefits it adds. Would any of that be a concern with integrating things here? I did stumble upon this issue on the xformers repo, but not sure if it would be the same/similar root cause as my friend described: xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219

[shirooo39]

ah... no wonder why I'm always getting exremely slightly different result from TheLastBen's implementation and my own simple implementation, even though I'm also using xformers

[MrKrzYch00]

When using xformers I changed "Random number generator source." from GPU to NV in settings and results became the same as when xformers were not used.
After changing it to CPU the result got the same (or very close to it) when I tested the AI with CPU only before, for exactly that purpose - comparing results.

So it seems that changing random generator source is all that is required to bring a specific behavior to result generation rather than switching between xformers or not or running the model on CPU specifically.

In other words, GPU RNG + disable-xformers behaved the same as NV RNG + xformers in my particular case - would need confirmation. Now the question would be, why using xformers cause GPU RNG to behave differently while not using them make it like NV RNG setting? Because like that, GPU setting seems somewhat like an undefined behavior (if it acts differently when using a slightly different platform setup while the hardware stays the same).