Configuring advanced datastream settings

Configuring advanced data collection settings

To configure advanced data collection settings for a datastream, follow these steps:

1. Go to the Datastreams page.

2. Open the chosen datastream by clicking on its name.

3. In the top navigation panel, click Local Data Retention.

4. Fill in the following fields:

   Extracts per fetch

   Select one of the following options:

   • Receive extracts as they are fetched - choose this option to keep the number of data extracts defined by the datastream configuration.

   • Split fetch into daily extracts - choose this option to create a separate data extract for each day in the time range specified for the fetch. The date is determined by the column set as a Key date column for the datastream. Use this option to ensure that the data collected by the datastream is distinct and there are no duplicate rows.

   Key Date Column

   Select the date column in your datastream.

   Retention type

   Specify how Adverity retains fetched data. Choose one of the following options:

   • To retain all data indefinitely, select Retain All.

   • To retain data for a specific number of fetches, select Retain N fetches and specify the number of fetches in the Retention Number field.

   • To retain data for a specific number of data extracts, select Retain N extracts and specify the number of data extracts in the Retention Number field.

   • To retain data for a specific number of days after it is fetched, select Retain N days and specify the number of days in the Retention Number field.

   Remove if empty

   Check this box to automatically delete data extracts without any rows when the fetch runs.

   Skip empty rows

   Select this bot to silently ignore empty rows in the extracts.

   Prune instantly

   Deleted data extracts are retained for a grace period by default. Check this box to permanently erase deleted data extracts without the usual grace period.

5. Click Save.

6. In the top navigation panel, click Settings.

7. Fill in the following fields:

   Fetch data sequentially (for advanced users only)

   Fetch jobs are executed simultaneously by default. Check this box to execute fetch jobs in sequential order, one after another.

   Share with children

   By default, child workspaces cannot access the datastreams of their parent workspace. Check this box to make this datastream available for all child workspaces of the workspace you currently use.

8. In the Monitors section, select the Add status column checkbox to include the status of the Data Quality monitors assigned to this datastream in the data extract.

   When this checkbox is selected, Adverity adds a new column to the data extract for each of the assigned monitors with the severity set to Warning. Status columns are named following this convention: dt_monitor_{{monitor's name}}_status.

   In Status for rows, select how the new column will show the monitor’s status for each data row from the following options:

   • as text (passed/failed)

   • as numbers (1/0)

9. Click Save.