Skip to content

Add support for FLUX ControlNet models (XLabs and InstantX) #7070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 125 commits into from
Oct 10, 2024
Merged

Add support for FLUX ControlNet models (XLabs and InstantX) #7070

merged 125 commits into from
Oct 10, 2024

Conversation

RyanJDick
Copy link
Contributor

@RyanJDick RyanJDick commented Oct 8, 2024 •
edited by hipsterusername
Loading

Summary

Add support for FLUX ControlNet models (XLabs and InstantX).

QA Instructions

  • SD1 and SDXL ControlNets, since the ModelLoaderRegistry calls were changed.
  • Single Xlabs controlnet
  • Single InstantX union controlnet
  • Single InstantX controlnet
  • Single Shakker Labs Union controlnet
  • Multiple controlnets
  • Weight, start, end params all work as expected
  • Can be used with image-to-image and inpainting.
  • Clear error message if no VAE is passed when using InstantX controlnet.
  • Install InstantX ControlNet in diffusers format from HF repo (InstantX/FLUX.1-dev-Controlnet-Union)
  • Test all FLUX ControlNet starter models

Merge Plan

No special instructions.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files python-tests PRs that change python tests labels Oct 8, 2024
@RyanJDick RyanJDick changed the title Add support for FLUX ControlNet models (XLabs and InstantX) Draft: Add support for FLUX ControlNet models (XLabs and InstantX) Oct 8, 2024
@RyanJDick RyanJDick changed the title Draft: Add support for FLUX ControlNet models (XLabs and InstantX) DRAFT: Add support for FLUX ControlNet models (XLabs and InstantX) Oct 8, 2024
@RyanJDick RyanJDick marked this pull request as draft October 8, 2024 21:57
@RyanJDick RyanJDick changed the title DRAFT: Add support for FLUX ControlNet models (XLabs and InstantX) Add support for FLUX ControlNet models (XLabs and InstantX) Oct 8, 2024
RyanJDick added 16 commits October 9, 2024 14:17
@RyanJDick
Add XLabs FLUX controlnet state dict key file to be used for developm…
2cc72b1 
…ent/testing.
@RyanJDick
Copy ControlNetFlux model from https://github.com/XLabs-AI/x-flux/blo…
2821ba8 
…b/47495425dbed499be1e8e5a6e52628b07349cba2/src/flux/controlnet.py.
@RyanJDick
Fix FLUX module imports for ControlNetFlux.
58eba8b
@RyanJDick
Remove duplicate FluxParams class.
9758e5a
@RyanJDick
Remove ControlNetFlux logic related to attn processor overrides.
eb5b662
@RyanJDick
Remove gradient checkpointing from ControlNetFlux.
9541156
@RyanJDick
Fix type errors and imporve docs for ControlNetFlux.
62d12e6
@RyanJDick
Add FLUX XLabs ControlNet model probing.
1d4a58e
@RyanJDick
First pass at integrating FLUX ControlNets into the FLUX Denoise invo…
c81bb76 
…cation.
@RyanJDick
Add support for FLUX controlnet weight, begin_step_percent and end_st…
36515e1 
…ep_percent.
@RyanJDick
Add InstantX FLUX ControlNet state dict for unit testing.
92b1515
@RyanJDick
Rename ControlNetFlux -> XLabsControlNetFlux
d1a0e99
@RyanJDick
Copy model from https://github.com/huggingface/diffusers/blob/99f6082…
9bcb93f 
…18caa069a2f16dcf9efab46959b15aec0/src/diffusers/models/controlnet_flux.py
@RyanJDick
Start updating imports for FluxControlNetModel
459cf52
@RyanJDick
Rename FluxControlNetModel -> DiffusersControlNetFlux
3578010
@RyanJDick
Remove logic for modifying attn processors from DiffusersControlNetFlux.
a17ea9b
RyanJDick and others added 20 commits October 10, 2024 07:59
@RyanJDick @hipsterusername
Update ControlNetCheckpointProbe.get_base_type() to work with InstantX.
ce4624f
@RyanJDick @hipsterusername
Bugfixes to get InstantX ControlNet working.
de414c0
@RyanJDick @hipsterusername
Create a dedicated FLUX ControlNet invocation.
e854181
@RyanJDick @hipsterusername
Add instantx_control_mode param to FLUX ControlNet invocation.
d99e7dd
@RyanJDick @hipsterusername
Shift the controlnet-type-specific logic into the specific ControlNet…
0559480 
… extensions and make the FLUX model controlnet-type-agnostic.
@RyanJDick @hipsterusername
(minor) Add comment about future memory optimization.
812940e
@RyanJDick @hipsterusername
Try to fix test failures affecting MacOS CI runners.
b3b7d08
@RyanJDick @hipsterusername
Revert "Try to fix test failures affecting MacOS CI runners."
118f0ba 
This reverts commit 216b36c.
@RyanJDick @hipsterusername
Skip tests that are failing on MacOS CI runners (for now).
8c12568
@RyanJDick @hipsterusername
Support installing InstantX ControlNet models from diffusers director…
63c4ac5 
…y format.
@RyanJDick @hipsterusername
Fix support for InstantX non-union models (with no single blocks).
908f656
@maryhipp @hipsterusername
update starter models to include FLUX controlnets
7daf3b7
@maryhipp @hipsterusername
update prepreprocessor logic to be more resilient
5fd3c39
@hipsterusername
ui: enable controlnet controls when FLUX is main model, update schema
eacdfc6
@hipsterusername
hide Control Mode for FLUX control net layer
5141be8
@RyanJDick @hipsterusername
Fix bug with InstantX input image range.
e7f9086
@hipsterusername
possibly a working FLUX controlnet graph
2bd4466
@RyanJDick @hipsterusername
Remove instantx_control_mode from FLUX ControlNet node.
a654dad
@RyanJDick @hipsterusername
Make FLUX controlnet node API more like SD API and get it working wit…
bb947c6 
…h linear UI.
@RyanJDick @hipsterusername
Reduce peak memory utilization when preparing FLUX controlnet inputs.
f37c501
@hipsterusername hipsterusername force-pushed the ryan/flux-controlnet-xlabs-instantx branch from 4aace24 to f37c501 Compare October 10, 2024 11:59
@hipsterusername
Copy link
Member

We should remove all of the starter controlnets and add the Shakker Pro Union.

Let's discuss the addition of control mode today - That seems to be the only controlnet that's worthwhile atm, so might want to make it as fully usable as we can

@RyanJDick RyanJDick merged commit 2d82e69 into main Oct 10, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/flux-controlnet-xlabs-instantx branch October 10, 2024 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maryhipp maryhipp maryhipp approved these changes

@hipsterusername hipsterusername hipsterusername approved these changes

@lstein lstein Awaiting requested review from lstein lstein is a code owner

@blessedcoolant blessedcoolant Awaiting requested review from blessedcoolant blessedcoolant is a code owner

@brandonrising brandonrising Awaiting requested review from brandonrising

@ebr ebr Awaiting requested review from ebr ebr is a code owner

@psychedelicious psychedelicious Awaiting requested review from psychedelicious psychedelicious is a code owner

Assignees
No one assigned
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@RyanJDick @hipsterusername @maryhipp