Queued experiments fail, can't see logs to understand what's up

[shcheklein]

Screen.Recording.2023-07-23.at.10.29.18.AM.mov

Single experiment runs fine even within a queue. Might be related to some resource allocation when it tries to do 2 of them, but I can't say what's going on.

It's related to me researching user complaining:

But if i run the same queue again..No metrics are getting displayed in Studio
Also these logged experiments are not getting deleted

[mattseddon]

Depends on iterative/dvc#9425

[shcheklein]

cc @dberenbaum

[dberenbaum]

@shcheklein Does dvc queue logs motor-abac work?

Edit: asking to see if it's only about iterative/dvc#9425 or if it is also related to iterative/dvc#9616.

[mjunker]

I run into the same issue. My impression is that this happens mainly when running lots of experiments (100 or more). I create experiments using the cli like this:

dvc exp run -S <several params with ranges> --queue --temp -f -n <name> dvc queue start -j 8

Also I do not get any output from dvc queue logs . When I apply the failed experiment to workspace it runs without any problem.

[shcheklein]

Making it p1 (even though it's blocked) since it comes my very often (cc @dberenbaum - let me know what you think). For the record we had also a new ticket #4524 , also I think I saw it today on Discord.

[dberenbaum]

The priority makes sense, but there are two underlying issues, and I'm not sure if we are trying to cover both here:

1. exp show: provide executor information for finished experiments dvc#9425: VS Code needs to detect which experiments have logs.
2. queue: log dvc errors dvc#9616: Queued experiments fail to generate logs if error happened before stage runs.