[RFC] [Semantic Convention] "Job" traces

[johnnyshields]

This PR defines a semantic convention for "Job" traces.

A "Job" models a batch job or task, which can be either enqueued, scheduled, or run on-demand.

Examples of job-related systems are:

• Sidekiq, instrumented here using "Messaging" convention
• Delayed Job, instrumented here using "Messaging" convention
• Celery, instrumented here using "Messaging" convention
• Resque, issue here
• Rake, issue here
• Apache Airflow, issue here

The spec for "Job" would be somewhere between "Messaging" and "FAAS". It would cover both the producer and consumer aspects of running jobs.

As noted above, OTEL instrumentation libraries are using the "Messaging" convention to represent Jobs today. However "Messaging", which is intended for Kafka, RabbitMQ, etc. is not a great fit:

• Messaging systems focus on the delivery of messages irrespective of the message contents. Jobs performing work/processing based on the job instruction.
• In a Messaging system the categorization is done by the "queue" or "topic" on which messages are transmitted; in a trace viewing system (e.g. Datadog, Lightstep, etc.) I would want to see messages labelled according to their queue. In Job systems, the categorization is based on job name/type, and the queue is just for worker (consumer) resource allocation. Hence I would want traces labelled as "ChargePaymentJob", "RefundPaymentJob", etc. even though all such jobs are in the same "payments" queue.

FAAS is also not ideal, as it is typically for serverless providers such as AWS Lambda which the infrastructure running the job has been abstracted away, and the primary focus is on the "function".

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ Johnny Shields (5208277)

[johnnyshields]

@dazuma @robertlaurin @adamantike

[iNikem]

I think we also need to do some modeling work. E.g. to answer the following questions:

• What SpanKind has the root span?
• If job executor reads a batch of tasks and splits it into chunks, should we have nested spans as well? Should they have the same semantic conventions? Should we have chunkId/itemId?
• How to decide when use messaging convention and when this new job? E.g. Kafka consumer listening a topic and then doing some job, is it messaging or job? Why?

[johnnyshields]

@iNikem

What SpanKind has the root span?

For DelayedJob, Rake, Airflow, etc. it would be SpanKind.CONSUMER for the process that performs jobs. Rake/Airflow run on-demand or by a timer aren't exactly "consumers" but it's the best fit out of all the existing span-kind values we have.

Any code that enqueues (and does not execute) the jobs would be SpanKind.PRODUCER.

If job executor reads a batch of tasks and splits it into chunks, should we have nested spans as well? Should they have the same semantic conventions? Should we have chunkId/itemId?

In practice, child jobs are typically enqueued as new jobs with new job_ids. It would be make more sense to do this as a parent_job_id attribute which would be common across all child jobs spawned from the original master job. Since each child is executed asynchronously in it's own process and thread they would not appear as "nested" with the parent. AFAIK DelayedJob/Resque/etc don't have such an attribute today but Airflow likely does as it is DAG-based.

How to decide when use messaging convention and when this new job? E.g. Kafka consumer listening a topic and then doing some job, is it messaging or job? Why?

A process which listens/consumes from Kafka and then performs a job synchronously is probably traced as a Messaging consumer because it's Kafka listening library will be instrumented. Theoretically you could also add in a Job span but it would be redundant with the message consumer.

[iNikem]

In practice, child jobs are typically enqueued as new jobs with new job_ids. It would be make more sense to do this as a parent_job_id attribute which would be common across all child jobs spawned from the original master job. Since each child is executed asynchronously in it's own process and thread they would not appear as "nested" with the parent. AFAIK DelayedJob/Resque/etc don't have such an attribute today but Airflow likely does as it is DAG-based.

I am more familiar with https://spring.io/projects/spring-batch which, it seems to me, follows different approach, hence my questions.

[iNikem]

Btw, @johnnyshields have you seen open-telemetry/semantic-conventions#1640 ?

[johnnyshields]

@iNikem no i had not seen that, funny because I searched a bit. I think it is roughly compatible with this one.

[johnnyshields]

@mateuszrzeszutek

[mateuszrzeszutek]

This roughly corresponds to the first two spans for spring-batch that I listed here.

• Is the CONSUMER/PRODUCER kind mandatory? This makes little sense for JSR-352 implementations (spring batch), as they do not concern themselves with queueing or scheduling, they just execute jobs in the current process (either synchronously or asynchronously). You can run trigger spring batch jobs with a HTTP call (button click on your website), and AFAIR this is what one of our clients does. Anyway, for job libraries that do not deal with queueing or messaging INTERNAL would be a better choice for the span kind.
• Is there any kind of result/status returned by at least one of those libs? Spring batch jobs return a user-defined exit_status string, which can be pretty much anything - still worth capturing.

If job executor reads a batch of tasks and splits it into chunks, should we have nested spans as well? Should they have the same semantic conventions? Should we have chunkId/itemId?

In practice, child jobs are typically enqueued as new jobs with new job_ids. It would be make more sense to do this as a parent_job_id attribute which would be common across all child jobs spawned from the original master job. Since each child is executed asynchronously in it's own process and thread they would not appear as "nested" with the parent. AFAIK DelayedJob/Resque/etc don't have such an attribute today but Airflow likely does as it is DAG-based.

This is rather different from what spring batch jobs look like. Spring batch jobs are hierarchical: every job is composed of steps, each step may be split into chunks (roughly equivalent to DB transactions), and each chunk may read, process and write several items. Everything in the same process.
Spring batch seems to be the only framework that structures its jobs this way though: I don't think that the general job spec should concern itself with steps and items, we should only model the job level here. Steps, chunks and items may be covered in a specific instrumentation spec (like the AWS one).

[mateuszrzeszutek]

I wonder if we need a small, separate semantic convention for scheduled operations: you can run a cron job (or use a ScheduledExecutorService in Java, or Quartz, or...) that runs any code, not necessarily a batch job or a FAAS function.

[johnnyshields]

I think this is the key question remaining for this RFC.

[johnnyshields]

Spring batch seems to be the only framework that structures its jobs this way though

I believe Apache Airflow is similar to Spring Batch in this regard. I guess there is a distinction between "Batch/Scheduled Jobs" and "Queued Jobs".

[Oberon00]

Re: #1582 (comment)

How to decide when use messaging convention and when this new job? E.g. Kafka consumer listening a topic and then doing some job, is it messaging or job? Why?

A process which listens/consumes from Kafka and then performs a job synchronously is probably traced as a Messaging consumer because it's Kafka listening library will be instrumented. Theoretically you could also add in a Job span but it would be redundant with the message consumer.

I also think this is an important question. We probably want to avoid cases where the sender uses messaging and the receiver job spans or vice versa. After looking at attributes I think that we should remove these:

• job.queue_name: messaging.destination
• net.peer.*: Also referenced by messaging.

If a job system uses queuing with a separate sender/producer of the jobs, messaging semantic conventions should be used additionally if further description of the queuing is desired.

Important editorial note: Please generate a markdown table in a new file under specification/trace/semantic_conventions/ and add a link in the README in that directory. Alternatively, you can add it to the same markdown file as messaging and change the title.
Adding a markdown file also gives you room for additional general explanation text.

[johnnyshields]

@Oberon00 I disagree that Job systems that have "queues" should use Messaging semantics; that's not a good litmus test. "Queue" in the context of a Jobs is usually an optional feature. For example, Ruby's Delayed Job will work all jobs in FIFO order, but it optionally allows you to set named "queues" and then assign worker resources to specific queues. Unlike a Messaging system where the queue/topic is the primary feature, in a Job system the queue is an optional/secondary attribute.

A better litmus test is, "does work happen based on the contents/payload of the message". Message system delivers the payload without looking inside; its job is simply to forward the message. A Job system does work based on the payload; a successfully worked Job is usually the end of the chain (unless a new/different message is generated as a result, e.g. an email sending job.)

In addition, net.peer.* should still be used here because ultimately traces are sent from physical hosts, and we want to see which host sent the trace.

[miketzian]

Ping so that this doesn't become stale, this looks like a useful addition

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[jmacd]

We can't merge this without more reviews, and I believe there is more work to do.

[ahayworth]

@ahayworth yes, I need help with the markdown documentation for this. I have some work in progress which I'll try to get committed this weekend.

@johnnyshields I can help with the documentation if needed - is there a preferred way you'd like to collaborate?

[johnnyshields]

@ahayworth I've naively committed all my work in progress on the docs. If you can just fork my PR and clean everything up that will be fine.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[simaoribeiro]

Hi @johnnyshields , are you playing to continue this work in the future?
As I mentioned in this discussion #2170, my team is currently migrating from OpenTracing to OpenTelemetry and a job scheduler specification is something that is missing.
My only experience is using Hangfire, but I would gladly help push this

[johnnyshields]

I'd be glad if someone can help take over this work to get to the finish line. Unfortunately I don't have the bandwidth for this at the moment.