Skip to content

feat(ui): canvas entity merging #7219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Oct 30, 2024
Merged

Conversation

psychedelicious
Copy link
Collaborator

@psychedelicious psychedelicious commented Oct 29, 2024

Summary

  • Refactor CanvasCompositorModule to be more flexible. The core functionality should be identical, but it now supports more types of compositing, as required by the new merge functions. Technically, it supports Merge Selected, but that is predicated on supporting multi-select on Canvas, which is out of scope for this PR.
  • Change Merge Visible behaviour to match raster editors. It creates a new layer, does not delete anything.
  • Add Merge Visible support for Regional Guidance. The masks are merged into a new RG layer with default RG settings.
  • Add Merge Visible support for Control Layers. The layers are merged using the "lighter" blend mode, which stacks up the lightness of every pixel. This leaves black pixels black.
  • Add Merge Down support for all layers. This works the same as in Affinity. The merged layers are discarded and an new layer created. Unlike Merge Visible, Merge Down merges disabled layers. RG and CL layers lose their settings. CL layers merge with the same blending as Merge Visible.
  • Added Save to Assets to the Inpaint Mask and Regional Guidance ctx menus. Not sure why this wasn't there before but as of this PR, all layer types have the same layer manipulation capabilities, so it makes sense to have the menu item.

Related Issues / Discussions

offline discussion

QA Instructions

  • Try Merge Down for each layer type
  • Try Merge Visible for Control Layers and Regional Guidance
  • Do some inpainting and outpainting (testing the core compositor module functionality)

There's one major question mark - the way Control Layers get merged.

  • Does the "lighter" blend mode do what we want? We can merge the layers according to any "composite operation", OR use a custom algorithm. For example, we could use the "transparency effect" algo before compositing, in which case two canny layers would merge to a transparent layer with white lines.
  • Do we want an option to merge them exactly the same as Raster Layers (stack them up without any blending)?

Merge Plan

n/a

Checklist

  • The PR has a short but descriptive title, suitable for a changelog

@github-actions github-actions bot added the frontend PRs that change frontend files label Oct 29, 2024
@hipsterusername
Copy link
Member

For the time being, lighter most likely serves the majority use case for merging. I think other layer merging can be explored as feedback rolls in

@hipsterusername
Copy link
Member

Approving but still need to test.

`CanvasCompositorModule` had a fairly inflexible API, only supporting compositing all raster layers or inpaint masks.

The API has been generalized work with a list of canvas entities. This enables `Merge Down` and `Merge Selected` functionality (though `Merge Selected` is not part of this set of changes).
Previously, merge visible deleted all other visible layers. This is not how affinity works, I should have confirmed before making it work like this in the first place.Ï
The "lighter" GlobalCompositeOperation is used. This seems to be the best one when merging control layers, as it retains edge maps.
@psychedelicious psychedelicious force-pushed the psyche/feat/ui/canvas-merge-down branch from 23582cc to ef5ee30 Compare October 29, 2024 22:25
Had missed several of these, which means we were invalidating caches far too often. For example, when you changed a RG prompt, we were invalidating the cached canvas for that entity, even though changing the prompt doesn't affect the canvas at all.
… calculation

Perf boost and reduces the number of images we create on the backend.
@psychedelicious psychedelicious force-pushed the psyche/feat/ui/canvas-merge-down branch from 530a8a4 to 2f1bcc4 Compare October 29, 2024 23:58
@psychedelicious
Copy link
Collaborator Author

Made some improvements the canvas' internal caching logic, as used by the compositor module. These changes improve UI performance and result in fewer images being created on the server:

  • Cache image data from composite layers (i.e. the actual pixel data used for generation mode calculation)
  • Cache transparency calculation results (i.e. the result of calculating the transparency of the image data for generation mode calculation)
  • Do not invalidate these caches when non-renderable state changes. For example, when an RG's prompt changes, we shouldn't invalidate the caches for its rendered mask, because the a change to the prompt never changes the mask.

@psychedelicious psychedelicious merged commit b091aca into main Oct 30, 2024
14 checks passed
@psychedelicious psychedelicious deleted the psyche/feat/ui/canvas-merge-down branch October 30, 2024 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
frontend PRs that change frontend files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants