ops: jobs: new jobs docs #784

yashlamba · 2025-06-19T13:35:32Z

Creating this PR to collaborate on Jobs documentation

* closes inveniosoftware#777

jobs: add how-to and explanation

mkdocs.yml

Co-authored-by: Carlin MacKenzie <[email protected]>

docs/maintenance/internals/jobs.md

docs/operate/customize/jobs.md

carlinmack

Looks good, a couple things to fix :)

carlinmack · 2025-06-20T13:04:16Z

docs/operate/ops/jobs/configure.md

+```
+
+### Scheduler
+Invenio Job System uses a custom celery task scheduler which requires you to run a separate - celery beat. You can run the custom beat in a separate container/shell by running:


Suggested change

Invenio Job System uses a custom celery task scheduler which requires you to run a separate - celery beat. You can run the custom beat in a separate container/shell by running:

The Invenio job system uses a custom celery task scheduler which requires a separate celery beat. You can run the custom beat in a separate container/shell like so:

kpsherva · 2025-06-20T14:38:22Z

docs/maintenance/internals/jobs.md

+
+## Why a Job System?
+
+Traditionally, Celery tasks in InvenioRDM were launched programmatically or via CLI. This created challenges:


this sentence is a bit strange to me, because usually we didn't run celery tasks by cli, in majority of cases celery tasks were launched by the application or cron

I agree that majority of cases but I have trigered a fet times the user_sync task via CLI, otherwise there was no other way.

docs/maintenance/internals/jobs.md

replace user with instance manager/developer. Co-authored-by: Karolina <[email protected]>

docs/maintenance/internals/jobs.md

kpsherva · 2025-06-20T14:49:04Z

docs/maintenance/internals/jobs.md

+
+## Logging & Auditability
+
+If a Celery task logs via `current_app.logger`, and is invoked as a job, its logs are captured and stored with the run metadata. These logs are shown in the admin UI. All logs are stored in OpenSearch and displayed via the job run UI and API.


this part looks to me more as a admin panel user documentation, not the maintenance description for jobs module. For maintenance it would be more relevant to explain how the log line ends up in the report (how to call logger and when - for different log levels)

kpsherva · 2025-06-20T14:50:29Z

docs/maintenance/internals/jobs.md

+- ⚠️ Partial Success (`TaskExecutionError`).
+- ❌ Failure (any other exception).
+
+Use `TaskExecutionError` when the task completes but with non-fatal issues (e.g., partial data processed).


general comment: when I read TaskExecutionError I expect failure and description in this sentence says it is used for non-fatal issues. I think it is confusing semantically

kpsherva · 2025-06-20T14:56:24Z

docs/maintenance/internals/jobs.md

+- Run immediately from the UI.
+- Scheduled with CRON or interval-based configurations via the admin panel.
+
+**Note:** There is no built-in concurrency lock. If the same job is triggered twice in parallel, both will run. Developers must ensure idempotency in task logic if required.


this should be highlighted in red, not only note

kpsherva · 2025-06-20T15:03:00Z

docs/maintenance/internals/jobs.md

+Jobs can be:
+
+- Run immediately from the UI.
+- Scheduled with CRON or interval-based configurations via the admin panel.


this seems to me also that could be a part of admin user documentation, not maintenance

kpsherva · 2025-06-20T15:03:36Z

docs/maintenance/internals/jobs.md

+
+---
+
+## Summary


do we need a summary?

kpsherva · 2025-06-20T15:06:42Z

docs/operate/customize/jobs.md

+
+---
+
+## Error Handling and Logging


this section should be next to https://github.com/inveniosoftware/docs-invenio-rdm/pull/784/files#diff-f746076d7b34f08dba7552507cae91027d192ff10bd6118a49b7e97d5261e205R17, I think it is more relevant to mention it there, while creating the task to remember to log things

kpsherva · 2025-06-20T15:14:29Z

docs/operate/ops/jobs/create_and_run.md

+
+- **Since:** Optional timestamp in `YYYY-MM-DD HH:mm` format to continue processing from that particular timestamp. _Uses the last successful run if left empty._
+
+#### Advanced Configuration


this part is only related with the datastream jobs, right? this is not made really clear IMO

kpsherva · 2025-06-20T15:18:34Z

docs/operate/ops/jobs/jobs.md

@@ -0,0 +1,5 @@
+## Jobs
+
+A powerful new job management system, giving administrators control over background tasks directly from the administration interface.


Suggested change

A powerful new job management system, giving administrators control over background tasks directly from the administration interface.

A powerful asynchronous tasks management interface called Invenio Jobs, giving administrators control over background tasks directly from the administration interface.

Co-authored-by: Karolina <[email protected]>

ops: jobs: new jobs docs

2b0e525

carlinmack added this to Sprint Q2/2025 🌻 Jun 19, 2025

carlinmack moved this to In progress in Sprint Q2/2025 🌻 Jun 19, 2025

0einstein0 linked an issue Jun 19, 2025 that may be closed by this pull request

Document and explain the new Jobs feature introduced in invenio-jobs #777

Open

jobs: updated docs for job runs through admin panel

a29d703

0einstein0 force-pushed the job-docs branch from 6f36923 to a29d703 Compare June 20, 2025 08:36

yashlamba and others added 5 commits June 20, 2025 10:53

ops: jobs: restructure and create docs

918ddf3

ops: jobs: spell check and links

39ba65b

jobs: add how-to and explanation

517b786

* closes inveniosoftware#777

jobs: add how-to and explanation

1988475

jobs: add how-to and explanation

maintainence: jobs: fix scheduler link

de948c3

yashlamba marked this pull request as ready for review June 20, 2025 09:32

carlinmack moved this from In progress to In review 🔍 in Sprint Q2/2025 🌻 Jun 20, 2025

carlinmack reviewed Jun 20, 2025

View reviewed changes

mkdocs.yml Outdated Show resolved Hide resolved

Update run -> Run

0f8490b

Co-authored-by: Carlin MacKenzie <[email protected]>

carlinmack reviewed Jun 20, 2025

View reviewed changes

docs/maintenance/internals/jobs.md Show resolved Hide resolved

carlinmack reviewed Jun 20, 2025

View reviewed changes

docs/operate/customize/jobs.md Show resolved Hide resolved

carlinmack reviewed Jun 20, 2025

View reviewed changes

docs/operate/customize/jobs.md Outdated Show resolved Hide resolved

carlinmack reviewed Jun 20, 2025

View reviewed changes

docs/operate/customize/jobs.md Show resolved Hide resolved

carlinmack reviewed Jun 20, 2025

View reviewed changes

docs/operate/customize/jobs.md Outdated Show resolved Hide resolved

carlinmack reviewed Jun 20, 2025

View reviewed changes

carlinmack moved this from In review 🔍 to In progress in Sprint Q2/2025 🌻 Jun 20, 2025

carlinmack reviewed Jun 20, 2025

View reviewed changes

jobs: improve docs

6f7d9ec

yashlamba moved this from In progress to In review 🔍 in Sprint Q2/2025 🌻 Jun 20, 2025

kpsherva reviewed Jun 20, 2025

View reviewed changes

docs/maintenance/internals/jobs.md Outdated Show resolved Hide resolved

Improve job docs

56231b7

replace user with instance manager/developer. Co-authored-by: Karolina <[email protected]>

kpsherva reviewed Jun 20, 2025

View reviewed changes

docs/maintenance/internals/jobs.md Outdated Show resolved Hide resolved

kpsherva reviewed Jun 20, 2025

View reviewed changes

docs/maintenance/internals/jobs.md

---

## Summary

Copy link

Contributor

kpsherva Jun 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need a summary?

kpsherva reviewed Jun 20, 2025

View reviewed changes

minor fix

381a5c4

Co-authored-by: Karolina <[email protected]>

	Invenio Job System uses a custom celery task scheduler which requires you to run a separate - celery beat. You can run the custom beat in a separate container/shell by running:
	The Invenio job system uses a custom celery task scheduler which requires a separate celery beat. You can run the custom beat in a separate container/shell like so:


		## Why a Job System?

		Traditionally, Celery tasks in InvenioRDM were launched programmatically or via CLI. This created challenges:


		## Logging & Auditability

		If a Celery task logs via `current_app.logger`, and is invoked as a job, its logs are captured and stored with the run metadata. These logs are shown in the admin UI. All logs are stored in OpenSearch and displayed via the job run UI and API.


		- Since: Optional timestamp in `YYYY-MM-DD HH:mm` format to continue processing from that particular timestamp. _Uses the last successful run if left empty._

		#### Advanced Configuration

		@@ -0,0 +1,5 @@
		## Jobs

		A powerful new job management system, giving administrators control over background tasks directly from the administration interface.

	A powerful new job management system, giving administrators control over background tasks directly from the administration interface.
	A powerful asynchronous tasks management interface called Invenio Jobs, giving administrators control over background tasks directly from the administration interface.

ops: jobs: new jobs docs #784

Are you sure you want to change the base?

ops: jobs: new jobs docs #784

Uh oh!

Conversation

yashlamba commented Jun 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carlinmack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kpsherva Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kpsherva Jun 20, 2025 •

edited

Loading