What has changed recently? Different and Worse Results from Same Settings

[tinrobot2000]

I've just had this issue where I have used the latest master [48dbf99] with the following settings
Prompt: "a photograph of an astronaut riding a horse" (the old chestnut eh?)
Steps: 50
Seed: 141382706
Size: 512 x 512
CFG: 7.5
Sampling: LMS
Weights: sd-v1-4.ckpt [7460a6fa]

Previous result from A1111 [Commit: 365d4b1] (InvokeAI produces almost identical result)

New result - note the banding at the top and bottom with the result not as resolved

Any thoughts?

[BunnyViking]

i agree something has changed recently... even running 1.4 pretty much everything is generating as if the word 'anthropomorphic' is in the prompt ... my monster prompts are generating humanoids when previously they were generating the appropriate animal most of the time.

[BunnyViking]

yup, ive set hypernetwork to 'none' just in case that was having an effect even when not using NAI (it was which im sure it didn't a few days ago) but the results are still far from the results I was getting with my old prompts.

[ciclon5]

Something weird just happened. i tried to downgrade back to 1.4 to see if it worked again but when i did. it didnt matter with hypernetwork i was using. it wasnt even taking my prompts at all. it was generating random stuff like forest, plains, logos and even football fields. when the prompt was something like "a male person wearing an office outfit in a hallway" or "a picture of an astronaut riding a horse" to try this case.

What in tarnation is going on with this model?. i think i might get out of collab all together and just run it locally later. i have the hardware

(in a few days i will try using NAI. i have tried 1.5 and now 1.4. if NAI its also changed and broken like this. i will assume the changes are intentional)

[gsgoldma]

None of my old prompts will produce the same image; this goes for all models. The results are all different and inferior.

[BunnyViking]

After spending a couple hours trying to get similar results to my old prompts I tent to agree. I've tried setting hypernetwork weight to 0, tried 1.4 and 1.5 with and without the .vae

I mean with 1.5 I don't expect the same results of course, it seems -very- different... but I assume 1.4 should behave the same as ever, I don't remember any patch notes saying anything had changed with seeds, and even still, the prompt should still give similar results with enough generations; I did up to 40 in testing and none were even close to the same style with the same prompt.

[ciclon5]

1.4 is broken for me too. as i said before in a prior comment. it doesnt even take in my prompts at all and generate whatever it wants apparently

[Gman0909]

I believe it has to do with xformers, the tradeoff for the increase in speed is a loss in accuracy, making replicating results from the same seed next to impossible.

[tinrobot2000]

I believe the problem is changing the model weights. I originally went from 1.4 -> 1.5 (banding appears) -> 1.4 (poor results)
Have run from the current head [6bd6154] and fresh load - everything is back to the way it was.
So suspect it is RAM issue see here #2180

[BunnyViking]

interesting .... it definitely feels like it's the weights that are screwed ... like doing 'skeletal wolf' just gave me a wolf with a couple bony bits... i had to do (skeletal:2) wolf to get an actual skeletal wolf (and many many other similar experiments with weights)

[BunnyViking]

Tried disabling xformers, and tried 1.4 after a reboot (so nothing should be cached) ... still poor results, old seeds dont work, and running their prompt doesn't give similar results.

[yankyankyank]

im also facing the same problem... i want to downgrade and go back to the previous version! i had really good result when i was using the "img2img alternative test" script now everything looks really bad even after multiple tries, checkpoint change ect...
any idea how to downgrade the UI? i want to go back to the version without all the "override sampling method" boxes.
thank you!

[tinrobot2000]

You can get a log of the master branch with git log --date=short --first-parent --pretty="%h %cd %s" which is a list of short SHAs, dates and commit subjects.
Once you have the commit hash you would like to use run git checkout <SHA hash>, this will set your repository to detached HEAD mode so you will not be on any branches
If you want to get up to date run git checkout master and then git pull

[dustinandrews]

Switching models works up to at least 6f7b7a3. Didn't have time to test past that.
'8aaadf5 2022-10-17 add cache for workflow' sounds maybe suspect based on the title. Cacheing is tricky sometimes.
For now I am just shutting down and moving models around by hand.

[tinrobot2000]

Manual for me too at the moment

[monophthongal]

Since you have a commit ID you know is good, a binary search over newer commits should be able to find the one that introduced the issue in just a few tries. (Pick a commit midway between the good commit and now and test it, pick a new midpoint before/after that one, repeat until you find the bug.)

[dustinandrews]

6f7b7a3 is last known good for me. Ran out of time to test further.

[Pernjakovic]

Similar thing here... I was training some embeddings since yesterday and several minutes ago just restarted the UI (with git pull). Every "new" embedding trained in the last 24 hours is wrong. Yet, they were spot on and working normally before I've restarted ui. No connection whatsoever with the trained material. I've pulled the latest updates and a minute later, nothing is working as it should - regarding weights. And trained embeddings. But I can use the ones trained several days ago, working normally. Go figure.

[smoke2007]

i'm having the same with hypernetworks, they simply don't converge anymore or burn out, it just keeps training endlessly 100.000 steps, without ever getting where it used to get before with 2000 steps.

The ones made before something changed are still fine. I've tried training with different rates for 16 hours, can't get it to work anymore, with multiple sets of images, nothing useable.

[Pernjakovic]

yup, one more thing, just trained the same embedding, same learning parameters, same training photos... not working as it should. Also, the images generated during training are mostly blurry and have some strange writings all over the place. And they do not produce a good final embedding. Not even close.

[ShinCoreSys]

I would caution against posting images of children, regardless of innocent intent.

[mykeehu]

I wrote about this problem here, and it seems that the training is becoming unstable because of the memory error.

[Pernjakovic]

I would caution against posting images of children, regardless of innocent intent.

Those are my kids, under my name and last name. Also, they are on various stock sites for over 14 years (getty, ss etc), all model released etc. Video and photos. And with SD they enjoy being ferries and princesses, zombies :) I understand your concern but also don't see anything questionable here. Except the gibberish and letters, frames and noise and blur that the SD is contributing to embeddings since yesterday/day before yesterday :)

[BunnyViking]

So i did a fresh install on another drive... old images arent generating exactly... but they are generating much more correctly... so there must be something that broke along the way in the past few days, something that didn't get updated right.

[BunnyViking]

and now after switching models a couple times we're back to trash again...

[nProtect]

Did you turn on option "Checkpoints to cache in RAM" ??

If yes, please change to 0 and test, This option has bug broken model when swapping ckpt.

[BunnyViking]

Did you turn on option "Checkpoints to cache in RAM" ??

If yes, please change to 0 and test, This option has bug broken model when swapping ckpt.

Nah I have it on 0, and I'm not using --xformers

It'd b great to know if something is getting cached, is there a way to tell, or a way to force clear it ?

[poudi7]

Has the problem been fixed? I haven’t updated because of this

[MrCheeze]

It would help a lot for someone who has the issue to do a binary search as mentioned above, to find the commit where things broke.

[mykeehu]

Here is the commit in which the reload has been added. Probably missing here is a part to remove embeddings (and maybe hypernetwork) from memory before loading, and only then reload. Before that, there was no problem because there was no reload, so the whole system had to be rebooted to load (so it was cleared from memory)

[poudi7]

Here is the commit in which the reload has been added. Probably missing here is a part to remove embeddings (and maybe hypernetwork) from memory before loading, and only then reload. Before that, there was no problem because there was no reload, so the whole system had to be rebooted to load (so it was cleared from memory)

That commit is 10 days old, isn’t this issue more recent? I believe the last time I updated was on the 21th, and I did a few test, I tried to reproduce something generated on the 13th and it is identical.

I also tried generating something from 3 week ago and the result was slightly different, but I believe that a few day before the 13th, new optimization were added, not xformer in my case because the improvements where to small with my 1070, but this one:
Applying cross attention optimization (Doggettx).

Now I haven’t used embeddings or hypernetwork, so that might be why I didn’t get different result.

[mykeehu]

I think that's where it started for me, because I was crazy about the feature at first and didn't pay attention to it, but when it started generating a lot of black and white images on one of my samples, that's where it got suspicious, so that's where I would start looking. It specifically started on October 18, so it's realistic. Commits between 16 and 18 October are worth looking at then.

[poudi7]

Have you tried deleting the hypernetwork and embedding to see if it change anything?

Also I just found this: https://reddit.com/r/StableDiffusion/comments/yfpgue/worse_results_with_new_a1111_cant_recreate_old/

the newer versions of automatic1111 the setting “Stop At last layers of CLIP model” was raised by 1. Setting it back to 1 produces the same output as before.

[smoke2007]

I think it's also the fact, that back in the beginning, create hypernetwork had nearly no options and choice, now it seems that there are endless lists of parameters you can change and the defaults are no longer there like "lineair" etc, and relu , swish, ... , I have no idea what these all do and I dont know which ones to select to have a hypernetwork that used to train correctly like before.

I did save some of my earlier hypernetworks before I started training on them, I might try and reuse one of those instead to start training from 0 again.

[R-N]

I dont know which ones to select to have a hypernetwork that used to train correctly like before.

I believe previously it was 1-2-1 linear, normal init, and without normalization nor dropout

[GigsTheCat]

This just started happening to me today in the most recent pull. With xformers enabled, pressing the generate button creates slightly different results each time, even with the same seed and settings. Example: https://www.youtube.com/watch?v=ToYd3yPyNWI

Disabling xformers fixed the issue for me.

[Rogal80]

141382706

I have very weird result with same settings on latest build - hmmm ??

[plscripts]

Confirm I got this result.

[mykeehu]

I've reported the bug, I don't know what's interfering with xformers

Update: "This is not a bug, this is a feature"

[0xdevalias]

A friend mentioned that using xformers could make things non-deterministic, and that there were a lot of references to it on the repo issues here. Wanting to understand a bit more about it, and link a bunch of potentially related issues together, I tried to find as many issues as I could that seemed to be related to xformers and the potential for it to be causing non-deterministic / unstable / inconsistent results:

The following may potentially be related (ordered by issue number):

Issues:

• Xformers optimization causes inconsistent generations with the same parameters #1999
• No token limit for prompts broken? Different results with the same prompts and seed #2005
• The same image can no longer be output under the same conditions. #2197
• On the problem that the same resolution can not be generated stably #2293
• Different results after swapping graphics card #2423
• Different images for same seed with xformers and "Highres. fix" enabled when ancestral samplers are used #2528
• Different outputs since WebUI update #2943
• xformers and euler_a variant results for same generation settings #3054
• [Bug]: The prompts do not produce the same results as before the latest updates on 10/22/22 #3476
• [Bug]: xformers always generate different results, seed is the same! #3967
• [Bug]: In img2img, setting denoising strength to 0 does not return original image #4011

Discussions:

• Different results after swapping graphics card #2418
• --xformers CMD line option causes slightly poorer quality images on GTX 1650 (Turing), possibly others? #2592
• Highres.fix composition #2705
• Exactly same prompt no more the same ? #2849
• xformers option has good and bad aspects #2958
• What has changed recently? Different and Worse Results from Same Settings #3461

I also came across this thread in the xformers repo, which while I can't guarantee is related, am wondering if it might be:

• xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219

And a question I raised on a PR in the diffusers repo:

• Up to 2x speedup on GPUs using memory efficient attention huggingface/diffusers#532 (comment)

  • A friend just mentioned that apparently using xformers tends to lower the quality of generated images, and apparently even causes StableDiffusion to generate different images for the same seed/settings. I haven't looked too deeply into things to try and validate that, but I was wondering if a) that is something that is currently known/documented, and b) if that's likely to be fixed at any point, or if it's just 'the price we pay' for the benefits it adds. Would any of that be a concern with integrating things here? I did stumble upon this issue on the xformers repo, but not sure if it would be the same/similar root cause as my friend described: xformers ViT-B ImageNet MAE + Deepnorm training instability facebookresearch/xformers#219

[zkx]

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在假期结束后，尽快给你回复。