Managing jobs in Management API

Listing all jobs in an instance

Use the API call documented in this section to list all currently running and scheduled jobs (for example, fetch jobs) in a whole instance.

To list all running and scheduled jobs, follow these steps:

1. Create a GET request to the following endpoint:

   https://{{INSTANCE}}/api/jobs
   

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/jobs/' \
--header 'Authorization: Bearer {{KEY}}'

The response to this request includes the state_label parameter with one of the following values assigned:

• Scheduled

• Running

• Retrying - the job is waiting out a retry/backoff. The underlying state remains Running while this label is shown; state_label is the only field that changes.

• Success

• Cancelled

• Failed

• Discarded - this status is set in the following cases:

  • If a preceding job failed or was cancelled, the follow up jobs are discarded.

  • If a job does not start running within 2 days of the fetch start, it is discarded.

  • If a job is cancelled manually, it is discarded.

• Stale - this status is not set currently. Generally, it can be used to label jobs that will not complete due to internal failures and lost or removed tasks.

The response of a fetch job also includes the user field:

The response also includes the last_executed field:

Each job’s nested datastream object also includes the auth and uuid fields:

Filtering the jobs

You can filter the jobs returned from this endpoint with a number of parameters:

For example, to list the jobs of a datastream, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?datastream_id={{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

For example, to list the jobs of datastreams in a specific workspace, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?stack_id={{WORKSPACE_ID}}' \
--header 'Authorization: Bearer {{KEY}}'

For example, to list the jobs within a given date range, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?start=YYYY-MM-DD&end=YYYY-MM-DD' \
--header 'Authorization: Bearer {{KEY}}'

For example, to list the jobs within a more specific time rangeby using timestamps, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?start=YYYY-MM-DDT00:00:00Z&end=YYYY-MM-DDT00:00:00Z' \
--header 'Authorization: Bearer {{KEY}}'

For example, to list the jobs of a certain type, for example Fetch, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?type=Fetch' \
--header 'Authorization: Bearer {{KEY}}'

For example, to list only the jobs triggered by a specific user, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?user=jane@doe.com' \
--header 'Authorization: Bearer {{KEY}}'

For example, to list only the jobs triggered by a person rather than by the system, import the following request example as raw text to your HTTP client (such as Postman):

curl --location -g --request GET 'https://{{INSTANCE}}/api/jobs/?triggered_by_user=true' \
--header 'Authorization: Bearer {{KEY}}'

Listing the fetch jobs of a specific datastream

To list the fetch jobs of a single datastream, you can also use the dedicated /api/datastreams/{{DATASTREAM_ID}}/jobs/ endpoint. This endpoint returns a paginated list of the datastream’s fetch jobs, ordered from the most recent to the oldest.

To list the fetch jobs of a datastream, follow these steps:

1. Create a GET request to the following endpoint:

   https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/jobs/
   

   Replace {{DATASTREAM_ID}} with the datastream’s ID.

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

As a result, you obtain a paginated list of the datastream’s fetch jobs.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location -g --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/jobs/' \
--header 'Authorization: Bearer {{KEY}}'

The response is a paginated envelope with the count, next, previous, and results fields. Use the page and page_size query parameters to page through the results:

curl --location -g --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/jobs/?page=1&page_size=50' \
--header 'Authorization: Bearer {{KEY}}'