feat: Set maximum job duration for Azure Batch

[adamrtalbot]

Adds a maximum job duration for Azure Batch, after which all jobs will terminate which will terminate them automatically and remove them from quota. This acts as a killswitch to prevent Job quota being used up if Nextflow dies unexpectedly.

Signed-off-by: adamrtalbot [email protected]

Hi! Thanks for contributing to Nextflow.

When submitting a Pull Request, please sign-off the DCO [1] to certify that you are the author of the contribution and you adhere to Nextflow's open source license [2] by adding a Signed-off-by line to the contribution commit message. See [3] for more details.

1. https://developercertificate.org/
2. https://github.com/nextflow-io/nextflow/blob/master/COPYING
3. https://github.com/apps/dco

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	6d3aa54
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/680a13aa6265cd0008e8d5a0

[bentsherman]

Why not just use the time directive?

[adamrtalbot]

Why not just use the time directive?

That is for a task not a job. Jobs are queues in Azure Batch and we keep leaving them hanging around forever until stuff breaks.

[pditommaso]

Is not the task already constrained by the container max wall time 👉 5c11a0d? in any case, agree with Ben, we should avoid introducing yet another setting

[adamrtalbot]

Is not the task already constrained by the container max wall time 👉 5c11a0d? in any case, agree with Ben, we should avoid introducing yet another setting

No, that is the time limit for the individual task not the job. Remember, a job is a queue in Azure Batch, which means Nextflow does not handle it in any way outside of closing it down when the pipeline closes. See this issue for details: #3926

This is a problem because there is a finite number of active jobs allowed by Azure Batch (e.g. maximum of 100). Azure seem to be becoming more strict with this which is causing problems and we need to be more frugal. When Nextflow exits abruptly, it leaves them in active state which uses up quota and requires them to be manually deleted.

By putting a finite time limit, orphaned jobs (queues) are correctly terminated after a block of time, which means they are much less likely to fill up the quota and prevent jobs and tasks being created.

[bentsherman]

Adam is right, queues in Azure are called "jobs", and apparently they aren't meant to be kept around after the workflow is finished. Still, I would try to find a better name for the setting because Nextflow users will be confused by maxJobTime vs the time directive

[adamrtalbot]

Adam is right, queues in Azure are called "jobs", and apparently they aren't meant to be kept around after the workflow is finished. Still, I would try to find a better name for the setting because Nextflow users will be confused by maxJobTime vs the time directive

Options:

• jobInactivityTimeout <- my favourite because it says what it's for not what it is
• jobMaxWallClockTime <- most accurate
• jobTimeoutLimit & maxJobDuration <- very similar to process.time

[bentsherman]

how about jobIdleTimeout

[pditommaso]

Java doc for task says:

Set the maxWallClockTime property: The maximum elapsed time that the Task may run, measured from the time the Task starts. If the Task does not complete within the time limit, the Batch service terminates it. If this is not specified, there is no time limit on how long the Task may run.

Java doc for job says:

Set the maxWallClockTime property: The maximum elapsed time that the Job may run, measured from the time the Job is created. If the Job does not complete within the time limit, the Batch service terminates it and any Tasks that are still running. In this case, the termination reason will be MaxWallClockTimeExpiry. If this property is not specified, there is no time limit on how long the Job may run.

I understand the rationale

When Nextflow exits abruptly, it leaves them in active state which uses up quota and requires them to be manually deleted.

However this may break long running jobs (tho 7 days could be quite unusual).

Enabling cleanup by default, would not improve the problem?

nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy

Lines 995 to 998 in de05ecc

	// delete all jobs
	if( config.batch().deleteJobsOnCompletion ) {
	cleanupJobs()
	}

[bentsherman]

The proposed timeout is for when a pipeline is killed before it can cleanup the jobs

[adamrtalbot]

Here's the refined version:

However, this may break long-running jobs (though 7 days could be quite unusual).

Correct, this is why I've set a long default value. It's essentially a dead man's switch, and we could also set it to 30, 60, or even 365 days. Setting it to null means the constraint won't be applied.

Wouldn't enabling cleanup by default improve the problem?

We already handle job termination and deletion effectively during normal pipeline runs, but we still encounter orphaned active jobs, typically when Nextflow is killed abruptly and isn't able to close properly. This change will cause jobs to terminate themselves after a period of inactivity, which means eventually you will return to zero active jobs.

[pditommaso]

Let's use jobMaxWallClockTime that consistent with Batch API with 30d as default

[adamrtalbot]

Let's use jobMaxWallClockTime that consistent with Batch API with 30d as default

Done and done.

[adamrtalbot]

This doesn't feel like a very good test...but I can't see how to mock the methods inside createJob0? Or is this an indication I should split up the method to a createJobConstraints and createJob0 method?

[adamrtalbot]

Done in a3dbc7c

[christopher-hakkaart]

Docs look great.

[pditommaso]

Using a static exception type is may be better, see for example

https://github.com/nextflow-io/nextflow/blob/master/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy#L873-L879