performance problems with dhall based generation

[uwedeportivo]

setup and steps to reproduce

• dhall version 1.35.0
• grab repo https://github.com/uwedeportivo/ds-dhall at branch main
• execute dhall --file ./src/generate-example.dhall in the root of the repo

observe: takes about 30 secs

now switch to the optimizations branch (i know, the irony :-) )

• git checkout optimizations
• execute dhall --file ./src/generate-example.dhall

it never returns. the difference between the two branches: in the "optimizations" branch the individual apply functions are a no-op identity function when the respective customization is not set in the overlay record. but their body has an additional merge expression. so not sure what causes the explosion. i did a binary search of how many of these individual functions i can apply before it starts hanging and it's about 4.

sample apply function in main branch:

let applyResourcesf0
    : baseSchema.Type → overlaySchema.Type → baseSchema.Type
    = λ(base : baseSchema.Type) →
      λ(overlay : overlaySchema.Type) →
        let baseResources =
              base.Precise-Code-Intel.Deployment.precise-code-intel-bundle-manager.spec.template.spec.containers.precise-code-intel-bundle-manager.resources

        let overlayResources =
              overlay.Precise-Code-Intel.Deployment.precise-code-intel-bundle-manager.containers.precise-code-intel-bundle-manager.resources

        let finalResources =
              resourceCombinator.overlayMerge baseResources overlayResources

        in  base
          with   Precise-Code-Intel
               . Deployment
               . precise-code-intel-bundle-manager
               . spec
               . template
               . spec
               . containers
               . precise-code-intel-bundle-manager
               . resources
               = finalResources

same function in the optimizations branch:

let applyResourcesf3
    : baseSchema.Type → overlaySchema.Type → baseSchema.Type
    = λ(base : baseSchema.Type) →
      λ(overlay : overlaySchema.Type) →
        let optionalOverlayResources =
              overlay.Precise-Code-Intel.Deployment.precise-code-intel-bundle-manager.containers.precise-code-intel-bundle-manager.resources

        let resultBase =
              merge
                { None = base
                , Some =
                    λ(overlayResources : resourceCombinator.Type) →
                      let baseResources =
                            base.Precise-Code-Intel.Deployment.precise-code-intel-bundle-manager.spec.template.spec.containers.precise-code-intel-bundle-manager.resources

                      let finalResources =
                            resourceCombinator.overlayMerge
                              baseResources
                              overlayResources

                      in  base
                        with   Precise-Code-Intel
                             . Deployment
                             . precise-code-intel-bundle-manager
                             . spec
                             . template
                             . spec
                             . containers
                             . precise-code-intel-bundle-manager
                             . resources
                             = finalResources
                }
                optionalOverlayResources

        in  resultBase

background about this repo

slides here: https://github.com/uwedeportivo/ds-dhall/blob/main/doc/slides.pdf. relevant repos: https://github.com/sourcegraph/ds-to-dhall, https://github.com/uwedeportivo/dhallie

[Gabriella439]

@uwedeportivo: I'll look into this more closely soon, but at the moment my focus is improving cache performance (which may or may not be related to this performance issue)

[uwedeportivo]

i want to be clear here that this is not urgent at all. it might be useful as an indicator of some performance bottleneck. on the other hand there might be simpler indicators.

[sjakobi]

• grab repo https://github.com/uwedeportivo/ds-dhall at branch main
• execute dhall --file ./src/generate-example.dhall in the root of the repo

I tried this with the current v1.38.0, but had to abort the command when dhall started using ~15GB of RAM! :/ I didn't even try the (de-)optimizations branch.