Creating and configuring scheduled fetches
On this page
This guide explains how to create a scheduled fetch and how to configure scheduled fetches, including custom schedules. To fetch data manually or view fetched data, see Fetching data.
Creating a scheduled fetch
To create a scheduled fetch, follow these steps:
-
Go to the Datastreams page.
-
In the Fetch section, turn on the Scheduling switch.
As a result, a default scheduled fetch is created using Smart Schedule.
By default, a datastream is scheduled with two ranges: a daily fetch that refreshes the last 3 days, and a weekly fetch that refreshes the last 40 days, giving you a frequent light-weight check and a wider periodic refresh out of the box. Datastreams backed by a file or datalake connector instead default to a single daily fetch that refreshes only the previous day. Both defaults run at a random time between 00:00 and 04:00 UTC (Coordinated Universal Time). This is because most data source APIs use UTC, which means that 00:00 UTC is the earliest time to import data for the previous day.
Configuring scheduled fetches
Configuring a custom scheduled fetch
After creating a scheduled fetch, you can configure it to collect data for your required time period at your required frequency. All schedules in the Custom Schedule of a datastream are managed together in the Set Custom Schedule dialog: add, edit, or remove any number of schedules there, then save them all at once. To configure a Custom Schedule, follow these steps:
-
Go to the Datastreams page.
-
In the Fetch section, choose one of the following options:
-
If this datastream already has a Custom Schedule, click Add/Edit to open the Set Custom Schedule dialog directly, then skip to the next step.
-
Otherwise, turn on the Scheduling switch, or click Add/Edit if Smart Schedule is already enabled. This opens the Set Smart Schedule dialog. Click Set to Custom Schedule to open the Set Custom Schedule dialog.
For a Bundle datastream, the Set Custom Schedule dialog offers two options, Smart Bundle and Custom Schedule. Select Custom Schedule. For more information about the Smart Bundle option, see Combining data extracts using Bundle.
-
-
In the Set Custom Schedule dialog, each existing schedule is shown as its own card. To add a new schedule, click Add schedule. If two schedules are identical, both cards show a Duplicate schedule tag, and Set schedule stays disabled until you change one of them. For each schedule you want to add or change, fill in the following fields:
- Frequency
Choose how often the fetch runs. If you choose Once, the scheduled fetch will be performed at the scheduled date and time, but will not repeat. A week is considered to run from Monday to Sunday, inclusive.
We refer to the day that the fetch is scheduled to run as the “scheduled date” in the following field descriptions.
- Day of week / Day of month
For Every week, Every 2 weeks, or Every 4 weeks, select one or more days of the week the fetch runs on. Use the Mon–Fri or Weekend quick-pick options, or select individual days.
For Every month, select the day of the month (1st–28th).
- Time range
Choose the time period for which Adverity collects data. Time presets work according to UTC.
To fetch data for the same dates each time the scheduled fetch runs, select Custom time range.
If you select Custom time range and leave the Date to field empty, Adverity dynamically sets this field to Yesterday.
To fetch all data available from the datastream each time the scheduled fetch runs, select Lifetime. This option is not available for all datastreams.
All the other options are relative to the scheduled date. For example:
Current month fetches data from the first day of the month, up to and including the scheduled date.
Previous month fetches data from the first to the last day of the month before the month the fetch runs.
Last 30 days fetches data for the 30-day period preceding, but not including, the scheduled date.
- Fetch start (UTC)
Choose the time when the fetch runs on the scheduled date. Specify the time according to the UTC (Coordinated Universal Time) standard.
- Date range offset (days)
(Optional) Specify the number of days by which Adverity shifts the fetched date range. A positive value shifts the date range into the past, a negative value shifts it into the future. Enter a value from -30 to 365. The offset changes only which dates are fetched, not when the fetch runs.
For example, if you select Yesterday in Time range and enter an offset of 1, a fetch that runs on March 10 collects data for March 8.
This field is not shown if you select Custom time range in Time range. In that case, the dates you select define the fetched date range.
-
(Optional) To remove a schedule, click Remove schedule on its card. You cannot remove the last remaining schedule.
-
Click Set schedule to save all schedules in the dialog at once.
As a result, Adverity saves every schedule in the dialog together, and the Custom Schedule takes effect for this datastream.
Example: Using a scheduled fetch to fetch the previous month’s data
You can use a scheduled fetch to collect data each month for the previous month. To do so, configure a schedule in the Set Custom Schedule dialog using these settings:
-
In Frequency, select Every month.
-
In Day of month, select 1.
-
In Fetch start (UTC), select the time in UTC that corresponds to 23:59 in your time zone.
-
In Time range, select Current month.
-
In Date range offset (days), enter 1.
As a result, each month, the fetch runs on the first day of the month, and the offset shifts the date range 1 day into the past. Adverity calculates Current month as if the fetch ran on the last day of the previous month, so it fetches data for all days of the previous month.
For example, when this fetch runs on April 1, Adverity collects data from March 1 to March 31.
Example: Using a scheduled fetch to collect data in a different time zone
Time presets in Adverity use UTC. In the following example, assume your account runs on Pacific Standard Time (PST) and you want to collect today’s data every 2 hours.
If you select Today in the Time range field, no data will be collected in any fetches between 00:00 and 08:00 UTC due to the difference between the time zones.
For example, if you fetch data at 02:00 UTC on July 7, at the time of the fetch, the local time for the data source is 18:00 PST on July 6. The fetch would attempt to collect data for a future date and would return no data.
To solve this problem, configure a schedule in the Set Custom Schedule dialog using these settings:
-
In Frequency, select Every 2 hours.
-
In Time range, select Last 2 Days. This option is available only for connectors that support it.
-
In Date range offset (days), enter -1.
As a result, the offset shifts the date range 1 day into the future, and Adverity fetches data for yesterday and today in UTC time. Until 08:00 UTC, it is effectively today and tomorrow in PST time. This means that the scheduled fetches return the most recent data, even when the dates for the two time zones are different.