Skip to content

FLUX Regional Prompting #7388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Nov 29, 2024
Merged

FLUX Regional Prompting #7388

merged 29 commits into from
Nov 29, 2024

Conversation

RyanJDick
Copy link
Contributor

@RyanJDick RyanJDick commented Nov 27, 2024

Summary

This PR adds support for regional prompting with FLUX.

Example 1

Global prompt: An architecture rendering of the reception area of a corporate office with modern decor.
image
image

QA Instructions

  • Test that there is no slowdown in the base case with a single global prompt.
  • Test image fully covered by regional masks.
  • Test image covered by region masks with small gaps.
  • Test region masks with large unmasked ‘background’ regions
  • Test region masks with significant overlap
  • Test multiple global prompts.
  • Test no global prompt.
  • Test regional negative prompts (It runs... but results are not great. Needs more tuning to be useful.)
  • Test compatibility with:
    • ControlNet
    • LoRA
    • IP-Adapter

Remaining TODO

  • Disable the following UI features for FLUX prompt regions: negative prompts, reference images, auto-negative.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

…stricted image self-attention and a mask with unrestricted image self attention.
@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files labels Nov 27, 2024
RyanJDick and others added 9 commits November 28, 2024 18:49
These helpers consolidate layer validation checks. For example, checking that the layer has content drawn, is compatible with the selected main model, has valid reference images, etc.
…n warnings

If there are any issues with the layer, the icon is displayed. If the layer is disabled, the icon is greyed out but still visible.
@psychedelicious
Copy link
Collaborator

psychedelicious commented Nov 29, 2024

Disabling specific features based on the selected model was getting a bit tricky. For example, there are many paths to creating a regional guidance reference image (IP Adapter), so it's hard to "block" all of those when FLUX is the selected model.

I've made a few changes to ease the dev burden and improve the user experience.

When a layer is invalid, we might disable the Invoke button and or skip that layer during graph building. For example, a raster layer is invalid when it has no objects in its canvas state, and a control layer is invalid when its ControlNet model is for a different architecture than the main model.

I've consolidated the validation logic into helper functions that return a list of warnings. These warnings are displayed in the layer list items. For example, when the selected model is FLUX and a negative prompt is used for a regional guidance layer:
image

The same helper functions are used for the warning tooltips, Invoke button "readiness" checks, and graph builders.

When a layer is disabled, the icon is greyed out but still visible:
image

When there are no warnings, there is no icon.

The warnings should be the same the warnings in the Invoke button tooltip, when it is disabled - the validation logic was mostly copied from there.

@RyanJDick RyanJDick merged commit 54b7f9a into main Nov 29, 2024
19 of 27 checks passed
@RyanJDick RyanJDick deleted the ryan/flux-regional-prompting branch November 29, 2024 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants