Implement 'Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models'

[0xdevalias]

Implement 'Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models'

Basically allows fine grained image tweaks using 'sliders'

• Paper: https://arxiv.org/abs/2311.12092

  • We present a method to create interpretable concept sliders that enable precise control over attributes in image generations from diffusion models. Our approach identifies a low-rank parameter direction corresponding to one concept while minimizing interference with other attributes. A slider is created using a small set of prompts or sample images; thus slider directions can be created for either textual or visual concepts. Concept Sliders are plug-and-play: they can be composed efficiently and continuously modulated, enabling precise control over image generation. In quantitative experiments comparing to previous editing techniques, our sliders exhibit stronger targeted edits with lower interference. We showcase sliders for weather, age, styles, and expressions, as well as slider compositions. We show how sliders can transfer latents from StyleGAN for intuitive editing of visual concepts for which textual description is difficult. We also find that our method can help address persistent quality issues in Stable Diffusion XL including repair of object deformations and fixing distorted hands. Our code, data, and trained sliders are available at this https URL

• Project Page: https://sliders.baulab.info/
• Code: https://github.com/rohitgandikota/sliders
• Weights: https://sliders.baulab.info/weights/xl_sliders/
• https://twitter.com/RohitGandikota/status/1727410973638852957
• https://www.reddit.com/r/StableDiffusion/comments/180zon7/concept_sliders_lora_adaptors_for_precise_control/

See Also

• [Feature Request]: Implement 'Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models' AUTOMATIC1111/stable-diffusion-webui#14064

[comfyanonymous]

They seem to work fine already? Here's a very quick test with eyesize.pt with strength_model -2.0 to strength_model 2.0

[rohitgandikota]

Hi team,

I'm Rohit- the author of "Concept Sliders." I would also strongly recommend using the inference time SDEdit technowue we use for more precise control. Basically, during inference, we use the pretrained model for the first t timesteps and then start using the sliders. This will allow for a smoother control.

[nomadoor]

Concept Sliders_eyesize.json

This could be an example of a workflow.
I used KSampler Advance with LoRA after 4 steps.

[guoreex]

This could be an example of a workflow. I used KSampler Advance with LoRA after 4 steps.

Great job, this is a method of using Concept Sliders with the existing LORA process. My understanding is that this method affects the computation of CLIP. However, it seems that the design of Concept Sliders is not equivalent to LORA. The author’s description is here: rohitgandikota/sliders#2 (comment). According to the author’s description, Concept Sliders can affect inference at multiple stages such as the text encoder, U-net, High res, etc. If we can precisely know that the source of the inference error is the text encoder or U-net, then we can use Concept Sliders to correct the inference at that stage without affecting other inference processes, precisely fixing errors. For example, the common problem with fingers, etc.

So what I want to express is, is there a way for Concept Sliders to access the computation at the U-net and more stage in comfyui?

An auto1111 webui plugin, LoRa Control - Dynamic Weights Controller (https://github.com/cheald/sd-webui-loractl#lora-control—dynamic-weights-controller), provides a control idea. The syntax used is <lora:network_name:0@0,1@0.5,0@1:unet=0@0,0.5@1>

I just started to get in touch with SD, I don’t know if my thoughts are accurate and reasonable, please correct me.

[nomadoor]

LoRa Control - Dynamic Weights Controller

If you prepare the same number of Ksampler Advanced nodes as the number of sampling steps, you can do the same kind of complex control of LoRA as this.
But I guess nobody would do that 🤣

Either way, since the inference methods seem to be different, a LoRA adapter dedicated to Concept Sliders needs to be developed.

[nomadoor]

Concept_Sliders_R.json
I had made a mistake in the settings of the KSamplerAdvanced node.
Although the difference is trivial, here is the corrected workflow.

• Previous Ver.
  -1st KSampler : return_with_leftover_noise - disable
  -2nd KSampler : add_noise - enable
• Current Ver.
  -1st KSampler : return_with_leftover_noise - enable
  -2nd KSampler : add_noise - disable

[brendanhoar]

IIRC, discussion about this elsewhere indicated that developers were waiting for resolution of #2666 , the execution model inversion changes, before attempting work like this.

Anyone want to take a look at to see if this is more feasible now?

[pecanela]

i'm really newbie about, but if I have many types of sliders for good eyes, correct hands, curlyhair, all archives in format .pt. Where directory should I put those archives in comfyUI folders?