Skip to content

LoRA refactor to enable FLUX control LoRAs w/ quantized tranformers #7446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 25 commits into from
Dec 17, 2024
Merged

LoRA refactor to enable FLUX control LoRAs w/ quantized tranformers #7446

merged 25 commits into from
Dec 17, 2024

Conversation

RyanJDick
Copy link
Contributor

@RyanJDick RyanJDick commented Dec 14, 2024 •
edited
Loading

Summary

This PR refactors the LoRA handling code to enable the use of FLUX control LoRAs on top of quantized transformers.

Changes:

  • Renamed a bunch of the model patching utilities to reflect that they are not LoRA-specific
  • Improved the unit test coverage.
  • Refactored the handling of 'sidecar' patch layers to make them work with more layer patch types. (This was necessary to get FLUX control LoRAs working on top of quantized models.)
  • Removed ONNXModelPatcher. It is out-of-date and hasn't been used in a while.

QA Instructions

I completed the following tests.

These should be repeated after changing the target branch to main.

Due to the large surface area of this PR, reviewers should do regression tests on a range of LoRA formats. There is a risk of regression on a specific format that was missed during the refactoring.

  • FLUX Control LoRA + full FLUX transformer
  • FLUX Control LoRA + BnB NF4 quantized transformer
  • FLUX Control LoRA + GGUF quantized transformer
  • FLUX Control LoRA + non-control LoRA + full FLUX transformer
  • FLUX Contro LoRA + non-control LoRA + BnB quantized transformer
  • FLUX Control LoRA + non-control LoRA + GGUF quantized transformer
  • Test the following cases for regression:
    • Misc SD1/SDXL LoRA variants (LoRA, LoKr, IA3)
    • FLUX, non-quantized, variety of LoRA formats
    • FLUX, quantized, variety of LoRA formats

Merge Plan

Don't merge this PR yet.

Merge plan:

  1. First merge brandon/flux-tools-loras into main
  2. Change the target branch of this PR to main
  3. Review / test / merge this PR

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files python-tests PRs that change python tests labels Dec 14, 2024
@brandonrising brandonrising force-pushed the ryan/lora-refactor-full branch from b9da053 to 54d5a46 Compare December 15, 2024 05:08
@RyanJDick RyanJDick mentioned this pull request Dec 17, 2024
Merged
14 tasks
@RyanJDick RyanJDick force-pushed the ryan/lora-refactor-full branch from 54d5a46 to a05642c Compare December 17, 2024 04:13
@RyanJDick RyanJDick marked this pull request as ready for review December 17, 2024 04:20
@RyanJDick RyanJDick mentioned this pull request Dec 17, 2024
Merged
6 tasks
@psychedelicious psychedelicious force-pushed the brandon/flux-tools-loras branch from 6c3281c to c358a63 Compare December 17, 2024 08:01
Base automatically changed from brandon/flux-tools-loras to main December 17, 2024 12:28
@RyanJDick
A unit tests for LinearSidecarWrapper (and fix a bug).
e2451ef
@RyanJDick
Add optimized implementations for the LinearSidecarWrapper when using…
1e0552c 
… LoRALayer or ConcatenatedLoRALayer patch types (since these are the most common).
@RyanJDick
Break up functions in LoRAPatcher in preparation for more refactoring.
ac28370
@RyanJDick
Switch LoRAPatcher to use the new sidecar_wrappers/ rather than sidec…
46133b5 
…ar_layers/.
@RyanJDick
Delete old sidecar_layers/ dir.
c76a448
@RyanJDick
Add sidecar wrapper for FLUX RMSNorm layers to support SetParameterLa…
606d58d 
…yers used by FLUX structural control LoRAs.
@RyanJDick
Ensure that patches are on the correct device when used in sidecar wr…
e7e3f7e 
…appers.
@RyanJDick
Move handling of LoRA scale and patch weight down into the layer patc…
fe09f2d 
…h classes.
@RyanJDick
Push LoRA layer reshaping down into the patch layers and add a new Fl…
37e3089 
…uxControlLoRALayer type.
@RyanJDick
Use a FluxControlLoRALayer when loading FLUX control LoRAs.
80f64ab
@RyanJDick
Add GGMLTensor op.
9369b39
@RyanJDick
Rename LoRAPatcher -> ModelPatcher.
c604a09
@RyanJDick
Rename ModelPatcher methods to reflect that they are general model pa…
b820862 
…tching methods and are not LoRA-specific.
@RyanJDick
Rename LoRAModelRaw to ModelPatchRaw.
7fad4c9
@RyanJDick
Rename ModelPatcher -> LayerPatcher to avoid conflicts with another M…
dd09509 
…odelPatcher definition.
@RyanJDick RyanJDick force-pushed the ryan/lora-refactor-full branch from c407a25 to dd09509 Compare December 17, 2024 13:21
@RyanJDick RyanJDick enabled auto-merge December 17, 2024 13:29
@RyanJDick RyanJDick merged commit 4d5f74c into main Dec 17, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/lora-refactor-full branch December 17, 2024 13:30
RyanJDick added a commit that referenced this pull request Dec 17, 2024
@RyanJDick
Add FLUX Control LoRA weight param (#7452)
594511c 
## Summary

Add the ability to control the weight of a FLUX Control LoRA.

## Example

Original image:
<div style="display: flex; gap: 10px;">
<img
src="https://github.com/user-attachments/assets/4a2d9f4a-b58b-4df6-af90-67b018763a38"
alt="Image 1" width="300"/>
</div>

Prompt: `a scarecrow playing tennis`
Weights: 0.4, 0.6, 0.8, 1.0
<div style="display: flex; gap: 10px;">
<img
src="https://github.com/user-attachments/assets/62b83fd6-46ce-460a-8d51-9c2cda9b05c9"
alt="Image 1" width="300"/>
<img
src="https://github.com/user-attachments/assets/75442207-1538-46bc-9d6b-08ac5c235c93"
alt="Image 2" width="300"/>
</div>
<div style="display: flex; gap: 10px;">
<img
src="https://github.com/user-attachments/assets/4a9dc9ea-9757-4965-837e-197fc9243007"
alt="Image 1" width="300"/>
<img
src="https://github.com/user-attachments/assets/846f6918-ca82-4482-8c19-19172752fa8c"
alt="Image 2" width="300"/>
</div>

## QA Instructions

- [x] weight control changes strength of control image
- [x] Test that results match across both quantized and non-quantized.

## Merge Plan

**_Do not merge this PR yet._**

1. Merge #7450 
2. Merge #7446 
3. Change target branch to main
4. Merge this branch.

## Checklist

- [ ] _The PR has a short but descriptive title, suitable for a
changelog_
- [ ] _Tests added / updated (if applicable)_
- [ ] _Documentation added / updated (if applicable)_
- [ ] _Updated `What's New` copy (if doing a release after this PR)_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hipsterusername hipsterusername hipsterusername approved these changes

@lstein lstein Awaiting requested review from lstein lstein is a code owner

@blessedcoolant blessedcoolant Awaiting requested review from blessedcoolant blessedcoolant is a code owner

@brandonrising brandonrising Awaiting requested review from brandonrising

@psychedelicious psychedelicious Awaiting requested review from psychedelicious psychedelicious is a code owner

@maryhipp maryhipp Awaiting requested review from maryhipp maryhipp is a code owner

Assignees
No one assigned
Labels
backend PRs that change backend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@RyanJDick @hipsterusername