Fix SDXL t2i adapters expect BGR instead of RGB #7205
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dunkeroni
requested review from
lstein,
blessedcoolant,
brandonrising,
RyanJDick,
hipsterusername and
psychedelicious
as code owners
github-actions
bot
added
python
psychedelicious
approved these changes
Oct 26, 2024
hipsterusername
approved these changes
Oct 26, 2024
3 tasks
psychedelicious
force-pushed
the
sdxl_t2i_bgr
branch
from
acfa9c8 to
3ff20bf
Compare
psychedelicious
added a commit
that referenced
this pull request
Nov 1, 2024
…7215) ## Summary This change mimics the unet padding strategy to align T2I featuremaps with the latents during denoising. It also slightly adjusts the crop and scale logic so that the control will match the input image without shifting when it needs to pad. ## Related Issues / Discussions <!--WHEN APPLICABLE: List any related issues or discussions on github or discord. If this PR closes an issue, please use the "Closes #1234" format, so that the issue will be automatically closed when the PR merges.--> ## QA Instructions Image generated at 1032x1024  Image generated at 1080x1040 to prove feature alignment.  Edge artifacts on the bottom and right are a result of SDXL's unet padding, and t2i influence will be cut off in those regions. ## Merge Plan Contingent on #7205 Currently the Canvas UI prevents users from generating non-64 resolutions while t2i adapter layers are active. Will leave this as a draft until fixing that. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Added conversions that trigger on SDXL to change the color order for t2i image inputs, which is required for OpenPose to work correctly. TencentARC t2i adapters were trained on cv2 preprocessor outputs in BGR format. As far as I can tell, those are the only publicly available SDXL-T2I available at this time, and OpenPose is the only one that is not greyscale. Changing the image order has no effect on canny/depth/etc.
Fix is applied for both main backend and the as-of-yet-unused modular backend. Also I snuck in a fix for the progress bar so the modular backend doesn't crash before denoising begins.
Related Issues / Discussions
https://www.reddit.com/r/StableDiffusion/comments/1f854k3/psa_fixing_sdxl_t2iadapter_openpose/
QA Instructions
Before:
After:
Merge Plan
Checklist