Skip to content

faster iteration over a Flatten of heterogenous iterators #58522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

adienes
Copy link
Member

@adienes adienes commented May 25, 2025

seems to help in many cases. would fix the precise MWE given in #52552, but does not necessarily fix comprehensively all perf issues of all heterogenous flattens. but, may as well be better when it's possible

setup:

julia> using BenchmarkTools

julia> A = rand(Int, 100000); B = 1:100000;

julia> function g(it)
           s = 0
           for i in it
               s += i
           end
           s
       end

before:

julia> @btime g($(Iterators.flatten((A, B))))
  12.461 ms (698979 allocations: 18.29 MiB)

julia> @btime g($(Iterators.flatten(i for i in (A, B))))
  12.393 ms (698979 allocations: 18.29 MiB)

julia> @btime g($(Iterators.flatten([A, B])))
  15.115 ms (999494 allocations: 25.93 MiB)

julia> @btime g($(Iterators.flatten((A, Iterators.flatten((A, B))))))
  82.585 ms (2997964 allocations: 106.78 MiB)

after:

julia> @btime g($(Iterators.flatten((A, B))))
  135.958 μs (2 allocations: 64 bytes)

julia> @btime g($(Iterators.flatten(i for i in (A, B))))
  149.500 μs (2 allocations: 64 bytes)

julia> @btime g($(Iterators.flatten([A, B])))
  17.130 ms (999498 allocations: 25.93 MiB)

julia> @btime g($(Iterators.flatten((A, Iterators.flatten((A, B))))))
  13.716 ms (398983 allocations: 10.67 MiB)

@adienes adienes added performance Must go faster iteration Involves iteration or the iteration protocol labels May 25, 2025
@adienes adienes added the potential benchmark Could make a good benchmark in BaseBenchmarks label Jun 3, 2025
@adienes
Copy link
Member Author

adienes commented Jun 17, 2025

@nanosoldier runbenchmarks("collection" || "scalar" || "array", vs=":master")

@adienes adienes removed the potential benchmark Could make a good benchmark in BaseBenchmarks label Jun 17, 2025
@nanosoldier
Copy link
Collaborator

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iteration Involves iteration or the iteration protocol performance Must go faster status: waiting for PR reviewer
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants