Thread-local current_task keeps garbage alive

[maleadt]

When returning values from tasks executed on another thread, those values are kept alive even though it should be possible to collect them:

julia> a = [2,2];
julia> finalizer(a) do _
         Core.println("finalizing a")
       end

julia> t = Threads.@spawn (println("returning a from thread $(Threads.threadid())"); a)
Task (runnable) @0x00007fffb73e8e80
returning a from thread 2

julia> dump(t)
Task
  next: Nothing nothing
  queue: Nothing nothing
  storage: Nothing nothing
  donenotify: Base.GenericCondition{Base.Threads.SpinLock}
    waitq: Base.InvasiveLinkedList{Task}
      head: Nothing nothing
      tail: Nothing nothing
    lock: Base.Threads.SpinLock
      owned: Int64 0
  result: Array{Int64}((2,)) [2, 2] #  <----------------- the array a is being kept alive by the task
  logstate: Nothing nothing
  code: #3 (function of type var"#3#4")
  _state: UInt8 0x01
  sticky: Bool false
  _isexception: Bool false

julia> wait(t)
julia> a = nothing
julia> t = nothing
julia> GC.gc(true)

# nothing is collected

Running under gdb reveals that the task object is being kept alive in the thread's local storage:

$ gdb --args julia -t2
(gdb) run

julia> code from above

(gdb) call jl_(jl_all_tls_states[1].current_task)
Task(next=nothing, queue=nothing, storage=nothing,
     donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.InvasiveLinkedList{Task}(head=nothing, tail=nothing), lock=Base.Threads.SpinLock(owned=0)),
     result=Array{Int64, (2,)}[2, 2], # <------------------- our array again
     logstate=nothing, code=Main.var"#3", _state=0x01, sticky=false, _isexception=false)

Running another task on the same thread replaces that sate and allows collection of the array:

julia> # code from above

julia> t = Threads.@spawn println("doing nothing from thread $(Threads.threadid())")
Task (runnable) @0x00007fffb5b99e40
doing nothing from thread 2

julia> GC.gc(true)
finalizing a

As observed in JuliaGPU/CUDA.jl#866.

[bkamins]

@quinnj - maybe this is what happens in CSV.jl?

[JosePereiraUA]

Also, maybe this is happening with PyCall + PyTorch ? (See https://discourse.julialang.org/t/using-gpu-via-pycall-causes-non-reusable-memory-allocation/55140)

[tkf]

I guess this is because we keep the current task during sleep? If so, a catch-all (and kinda horrible) workaround may be to run

Threads.@threads :static for _ in 1:Threads.nthreads()
    Timer(Returns(nothing), 0; interval = 1)
end

?

Now that we store Task in the actual TLS and not ptls, I think we can't GC the current task even when the worker thread sleeps. We also can't unconditionally clear out jl_current_task->result = NULL since there may be references to this task (and fetch can be called). Several options I can think of:

1. Create a dummy task per worker thread and set it as the current task before sleep.
2. Minimize Core.Task fields (without result field and also other auxiliary things) and wrap it in Base.Task and use it in Threads.@spawn etc. We can then GC the Base.Task while keeping Core.Task alive (via ptls).

For a short-term solution with relatively small changes, I think Option 1 is attractive. Option 2 is attractive from other points of view like task fusion where you'd want to create tasks without materializing the auxiliary fields (e.g., RNG).

[pxl-th]

Bumped into this issue with JuliaGPU/AMDGPU.jl#299, where each ROCKernelSignal spawns waiter on a separate thread, but after it finishes it prevents finalizers frome being called (and thus signals are not deleted until Julia exits, same for ROCQueue).

Proposed workaround in JuliaGPU/CUDA.jl#866 does not work as those threads do not return.

Here's also a small MWE that reflects AMDGPU's situation.

[bkamins]

bump

[bkamins]

Maybe the simplest solution would be to make sure that then GC is invoked the solution proposed by @tkf is invoked before GC process? (the point is that, AFAICT, GC currently ensures that it does not run in parallel to anything else - but I do not know enough about details of current GC implementation so this might be a wrong approach)