Scheduling fetches in Management API

On this page

This guide explains how to configure custom cron-based fetch schedules for your datastreams using the Management API.

See PATCH /api/datastreams/{id}/ and PATCH /api/v1/datastreams/ in the API reference.

Introduction

Scheduling lets you configure fetches that periodically retrieve data extracts from your datastreams.

Prerequisites

Before you complete the procedure in this guide, perform all of the following actions:

Use a Connect API key with the following permissions:
- Datastream: Read-only (datastream:read) to retrieve schedules.
- Datastream: Write (datastream:write) to change schedules.
Picking Write for a permission also gives the key its Read-only access. Without the permission a request needs, Management API returns HTTP 403. You cannot change the permissions of a key, so to add one, create a new key. For more information, see Creating a Connect API key in Adverity.
Create a datastream and determine its datastream ID. For more information, see Creating datastreams.

Retrieving existing schedules

To retrieve existing schedules, follow these steps:

Create a GET request to the following endpoint:

https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/schedules

In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, you obtain the list of existing schedules for the datastream. Search for schedules in the response to find currently configured schedules.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location -g --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/schedules' \
--header 'Authorization: Bearer {{KEY}}'

Retrieving schedule and date range options

To retrieve schedule options and date range options for a particular datastream, follow these steps:

Create an OPTIONS request to the following endpoint:

https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/

In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
Send the request.

As a result, you obtain the available schedule and date range options for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location -g --request OPTIONS 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

Example of the part of response with two options for fetch date range (choices) of the time_range_preset variable:

"time_range_preset": {
   "type": "choice",
   "required": false,
   "read_only": false,
   "label": "Time range preset",
   "help_text": "Select one of the time range presets.",
   "choices": [
        {
          "display_name": "Fixed Time Range",
          "value": 0
        },
        {
          "display_name": "Yesterday",
          "value": 13
        },

Configuring a schedule and date range

After you configured an OPTIONS call to retrieve the accepted parameters for schedule options, you can edit these parameters. To configure a schedule, follow these steps:

Create a PATCH request to the following endpoint:

https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/

In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.
In the HTTP request body, include the parameter time_range_preset. Set the value of this parameter according to the options retrieved in the API call described in the previous section. This parameter specifies the date range of the fetch (for which days, weeks, and other time specific values you are requesting the data).
In the HTTP request body, include the parameter and value "cron_type": "week". This parameter specifies how often the fetch is periodically scheduled. "cron_type": "week" fetches the data of a datastream once per week.
Send the request.

As a result, the schedule and date range are configured for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "time_range_preset": 11,
    "cron_type": "week"
}'

Creating a new schedule

To create a new schedule, send a PATCH request that includes all required variables for a fetch schedule. Find the necessary values using the OPTIONS request described in section Retrieving schedule options above. To create a new schedule, follow these steps:

Create a PATCH request to the following endpoint:

https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/

In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.

In the HTTP request body, include the following parameters and values wrapped in a schedules array (edit them as necessary):

{"schedules": [
    {
        "cron_preset": "CRON_EVERY_DAY",
        "cron_type": "day",
        "cron_interval": 1,
        "cron_start_of_day": "00:00:00",
        "cron_interval_start": 1,
        "time_range_preset": 0,
        "time_range_preset_label": "Fixed Time Range",
        "delta_type": 1,
        "delta_interval": 1,
        "delta_start_of_day": "00:00:00",
        "delta_interval_start": 1,
        "fixed_start": "2021-12-01",
        "fixed_end": "2021-12-02",
        "offset_days": 0
    }
]}

Send the request.

As a result, the new schedule is created for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"schedules": [
{
    "cron_preset": "CRON_EVERY_DAY",
    "cron_type": "day",
    "cron_interval": 1,
    "cron_start_of_day": "00:00:00",
    "cron_interval_start": 1,
    "time_range_preset": 0,
    "time_range_preset_label": "Fixed Time Range",
    "delta_type": 1,
    "delta_interval": 1,
    "delta_start_of_day": "00:00:00",
    "delta_interval_start": 1,
    "fixed_start": "2021-12-01",
    "fixed_end": "2021-12-02",
    "offset_days": 0
}
]}'

Creating multiple schedules

Similarly to the procedure described in the previous section, send a PATCH request that includes all required variables for multiple fetch schedules. Find the necessary values using the OPTIONS request described in section Retrieving schedule options above. To create multiple schedules, follow these steps:

Create a PATCH request to the following endpoint:

https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/

In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.

In the HTTP request body, include the following parameters and values wrapped in a schedules array (edit them as necessary):

{"schedules": [
    {
        "cron_preset": "CRON_EVERY_DAY",
        "cron_type": "day",
        "cron_interval": 1,
        "cron_start_of_day": "00:00:00",
        "cron_interval_start": 1,
        "time_range_preset": 0,
        "time_range_preset_label": "Fixed Time Range",
        "delta_type": 1,
        "delta_interval": 1,
        "delta_start_of_day": "00:00:00",
        "delta_interval_start": 1,
        "fixed_start": "2021-12-01",
        "fixed_end": "2021-12-02",
        "offset_days": 0
    },
    {
        "cron_preset": "CRON_EVERY_WEEK",
        "cron_type": "week",
        "cron_interval": 1,
        "cron_start_of_day": "00:00:00",
        "cron_interval_start": 1,
        "time_range_preset": 0,
        "time_range_preset_label": "Fixed Time Range",
        "delta_type": 1,
        "delta_interval": 1,
        "delta_start_of_day": "00:00:00",
        "delta_interval_start": 1,
        "fixed_start": "2021-12-01",
        "fixed_end": "2021-12-02",
        "offset_days": 0
    }
]}

Send the request.

As a result, the schedules are created for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"schedules": [
{
    "cron_preset": "CRON_EVERY_DAY",
    "cron_type": "day",
    "cron_interval": 1,
    "cron_start_of_day": "00:00:00",
    "cron_interval_start": 1,
    "time_range_preset": 0,
    "time_range_preset_label": "Fixed Time Range",
    "delta_type": 1,
    "delta_interval": 1,
    "delta_start_of_day": "00:00:00",
    "delta_interval_start": 1,
    "fixed_start": "2021-12-01",
    "fixed_end": "2021-12-02",
    "offset_days": 0
},
{
    "cron_preset": "CRON_EVERY_WEEK",
    "cron_type": "week",
    "cron_interval": 1,
    "cron_start_of_day": "00:00:00",
    "cron_interval_start": 1,
    "time_range_preset": 0,
    "time_range_preset_label": "Fixed Time Range",
    "delta_type": 1,
    "delta_interval": 1,
    "delta_start_of_day": "00:00:00",
    "delta_interval_start": 1,
    "fixed_start": "2021-12-01",
    "fixed_end": "2021-12-02",
    "offset_days": 0
}
]
}'

Using smart schedule

Smart schedule is an automated scheduling mode that handles fetch timing intelligently. Instead of defining a fixed cron expression, smart schedule automatically determines when to fetch data for each datastream, taking into account factors such as API rate limits, datastream dependencies, and historical processing times. For more information about smart schedule, see Using Smart Schedule.

You can switch a datastream between smart schedule and custom (cron-based) schedule using PATCH /api/v1/datastreams/ with the datastream_id query parameter.

Enabling smart schedule

To enable smart schedule on a datastream, follow these steps:

Create a PATCH request to the following endpoint, replacing {{DATASTREAM_UUID}} with the UUID of the datastream:
```
https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.

In the HTTP request body, set pulling.active_mode to "smart":

{
    "pulling": {
        "active_mode": "smart"
    }
}

Send the request.

As a result, smart schedule is enabled for the selected datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"pulling": {"active_mode": "smart"}}'

A successful request returns a 200 response listing the affected datastreams:

{
    "affected": {
        "datastream_ids": ["{{DATASTREAM_UUID}}"]
    }
}

Switching back to a custom schedule

To switch a datastream back to a custom (cron-based) schedule, follow these steps:

Create a PATCH request to the following endpoint, replacing {{DATASTREAM_UUID}} with the UUID of the datastream:
```
https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}
```
In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.
In the HTTP request header, include the parameter Content-Type with value application/json.

In the HTTP request body, set pulling.active_mode to "schedule":

{
    "pulling": {
        "active_mode": "schedule"
    }
}

Send the request.

As a result, the datastream switches to a custom (cron-based) schedule.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/v1/datastreams/?datastream_id={{DATASTREAM_UUID}}' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"pulling": {"active_mode": "schedule"}}'

After switching back to a custom schedule, configure the cron fields (such as cron_type, cron_interval, and cron_start_of_day) using the procedure described in Configuring a schedule and date range.

Filtering datastreams by schedule type

The GET /api/v1/datastreams/ endpoint supports filtering by active pulling mode. Use the active_pulling_mode query parameter with a value of "smart" or "schedule" to list only datastreams using that mode. For the full list of filter parameters, refer to the interactive API documentation at https://{{INSTANCE}}/api/v1/docs/.