Throwing errors after 1000 time steps simulation

[ErikQ97]

Hi there,

I am running a simulation with Trixi.jl and I want to obtain the gradients of the parameters. When I took different trials I found that the algorithm works well when the total time step is less than 1000, however, if the total time steps exceed 1000 there would be an error throwed as

ERROR: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{typeof(predict), Float64}, Float64, 1}) The type Float64 exists, but no method is defined for this combination of argument types when trying to construct it.

Any comments or suggestions would be highly appreciated.

Thank you!

[KristofferC]

See https://juliadiff.org/ForwardDiff.jl/dev/user/limitations/.

The code has to be generic, apparently it does something on the 1000 step which is not. Impossible to say what with the information provided here.

[mcabbott]

The authors of Trixi.jl may be able to say something, if you post the entire stack trace -- not just the first line.

The error shown is (as Kristoffer says) the usual error when you try to write a Dual number into a non-Dual container. But someone needs to figure out why exactly something is doing that at the 1000th step, and knowing what function is being called may help.

[ErikQ97]

Thanks for the reply!

The entire trace back is as follow:

ERROR: MethodError: no method matching Float64(::ForwardDiff.Dual{ForwardDiff.Tag{typeof(predict), Float64}, Float64, 1})
The type `Float64` exists, but no method is defined for this combination of argument types when trying to construct it.

Closest candidates are:
  (::Type{T})(::Real, ::RoundingMode) where T<:AbstractFloat
   @ Base rounding.jl:265
  (::Type{T})(::T) where T<:Number
   @ Core boot.jl:900
  Float64(::IrrationalConstants.Logten)
   @ IrrationalConstants ~/.julia/packages/IrrationalConstants/lWTip/src/macro.jl:131
  ...

Stacktrace:
  [1] convert(::Type{Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{typeof(predict), Float64}, Float64, 1})
    @ Base ./number.jl:7
  [2] convert(::Type{Union{Nothing, Float64}}, x::ForwardDiff.Dual{ForwardDiff.Tag{typeof(predict), Float64}, Float64, 1})
    @ Base ./some.jl:37
  [3] Progress
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:100 [inlined]
  [4] Progress
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:104 [inlined]
  [5] #_asprogress#5
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:183 [inlined]
  [6] _asprogress (repeats 2 times)
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:175 [inlined]
  [7] #asprogress#4
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:168 [inlined]
  [8] (::VSCodeServer.var"#62#63"{@Kwargs{…}, VSCodeServer.var"#57#58", Tuple{…}, Module})()
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.127.2/scripts/packages/VSCodeServer/src/progress.jl:73
  [9] #invokelatest#2
    @ ./essentials.jl:1055 [inlined]
 [10] invokelatest
    @ ./essentials.jl:1052 [inlined]
 [11] #try_process_progress#61
    @ ~/.vscode/extensions/julialang.language-julia-1.127.2/scripts/packages/VSCodeServer/src/progress.jl:72 [inlined]
 [12] try_process_progress
    @ ~/.vscode/extensions/julialang.language-julia-1.127.2/scripts/packages/VSCodeServer/src/progress.jl:68 [inlined]
 [13] handle_message(j::VSCodeServer.VSCodeLogger, level::Base.CoreLogging.LogLevel, message::String, _module::Module, group::Symbol, id::Symbol, file::String, line::Int64; kwargs::@Kwargs{…})
    @ VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.127.2/scripts/packages/VSCodeServer/src/progress.jl:9
 [14] #invokelatest#2
    @ ./essentials.jl:1057 [inlined]
 [15] invokelatest
    @ ./essentials.jl:1052 [inlined]
 [16] macro expansion
    @ ./logging/logging.jl:388 [inlined]
 [17] log_step!(progress_name::String, progress_id::Symbol, progress_message::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), dt::ForwardDiff.Dual{…}, u::Vector{…}, p::SemidiscretizationHyperbolic{…}, t::ForwardDiff.Dual{…}, tspan::Tuple{…})
    @ OrdinaryDiffEqCore ~/.julia/packages/OrdinaryDiffEqCore/yTEch/src/integrators/integrator_utils.jl:283
 [18] _loopfooter!(integrator::OrdinaryDiffEqCore.ODEIntegrator{…})
    @ OrdinaryDiffEqCore ~/.julia/packages/OrdinaryDiffEqCore/yTEch/src/integrators/integrator_utils.jl:257
 [19] loopfooter!
    @ ~/.julia/packages/OrdinaryDiffEqCore/yTEch/src/integrators/integrator_utils.jl:207 [inlined]
 [20] solve!(integrator::OrdinaryDiffEqCore.ODEIntegrator{…})
    @ OrdinaryDiffEqCore ~/.julia/packages/OrdinaryDiffEqCore/yTEch/src/solve.jl:621
 [21] #__solve#62
    @ ~/.julia/packages/OrdinaryDiffEqCore/yTEch/src/solve.jl:7 [inlined]
 [22] __solve
    @ ~/.julia/packages/OrdinaryDiffEqCore/yTEch/src/solve.jl:1 [inlined]
 [23] solve_call(_prob::ODEProblem{…}, args::SSPRK33{…}; merge_callbacks::Bool, kwargshandle::Nothing, kwargs::@Kwargs{…})
    @ DiffEqBase ~/.julia/packages/DiffEqBase/HGITF/src/solve.jl:635
 [24] solve_call
    @ ~/.julia/packages/DiffEqBase/HGITF/src/solve.jl:592 [inlined]
 [25] #solve_up#44
    @ ~/.julia/packages/DiffEqBase/HGITF/src/solve.jl:1128 [inlined]
 [26] solve_up
    @ ~/.julia/packages/DiffEqBase/HGITF/src/solve.jl:1106 [inlined]
 [27] #solve#42
    @ ~/.julia/packages/DiffEqBase/HGITF/src/solve.jl:1043 [inlined]

Will this be a bit more helpful for debugging?

[mcabbott]

The proximate cause seems to be logging code, and you might be able to avoid this by e.g. https://docs.julialang.org/en/v1/stdlib/Logging/#Logging.disable_logging

You might need to ask people involved with ProgressLogging or DiffEqBase, or Trixi. Posting on https://discourse.julialang.org/ might get it seen by someone. I think it's very unlikely that the fault is with this package, ForwardDiff.

[KristofferC]

[17] log_step!(progress_name::String, progress_id::Symbol, progress_message::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), dt::ForwardDiff.Dual{…}, u::Vector{…}, p::SemidiscretizationHyperbolic{…}, t::ForwardDiff.Dual{…}, tspan::Tuple{…})

dt here is a dual which apparently cannot be passed to the logging system at https://github.com/SciML/OrdinaryDiffEq.jl/blob/2d62c865a96a50e53f245bb12e1b461f179d2d4b/lib/OrdinaryDiffEqCore/src/integrators/integrator_utils.jl#L281-L286.

Closing since this isn't really anything that can be fixed on the ForwardDiff side.