Configuring datastreams in Management API

Prerequisites

Before you complete the procedures in this guide, perform the following action:

• Use a Connect API key with the following permissions:

  • Datastream: Read-only (datastream:read) to list and view datastreams.

  • Datastream: Write (datastream:write) to update or delete datastreams.

  Picking Write for a permission also gives the key its Read-only access. Without the permission a request needs, Management API returns HTTP 403. You cannot change the permissions of a key, so to add one, create a new key. For more information, see Creating a Connect API key in Adverity.

Listing datastreams of a specific type

To list datastreams of a specific datastream type, follow these steps:

1. Create a GET request to the following endpoint:

   https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/
   

   Replace {{DATASTREAM_TYPE}} with the datastream type’s ID. For more information on determining a datastream type’s ID, see Determining a Datastream Type’s ID.

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Optionally, add query parameters to filter the results:

   Scroll horizontally to see the full table.

   Parameter	Description
   auth	One or more authorization IDs. Returns only the datastreams that use one of the given authorizations. Separate multiple values with commas, for example auth=1,2,3.
   creator	The username of the datastream’s creator.
   ids	One or more datastream IDs. Separate multiple values with commas, for example ids=1,2,3.
   name	Filter by datastream name. Matches names that contain this value.
   stack_id	The ID of the workspace the datastream belongs to.
   transformer	The ID of a transformation assigned to the datastream.
   scheduled_deletion	Filter by scheduled deletion state. Use true to return only the datastreams that are scheduled for deletion, or false to exclude them. Omit the parameter to return both.

4. Send the request.

As a result, you obtain the list of datastreams of a specific datastream type.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/' \
--header 'Authorization: Bearer {{KEY}}'

Example of JSON response:

{
    "count": 15,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": {{DATASTREAM_ID}},
            "uuid": "{{DATASTREAM_UUID}}",
            "cron_type": "day",
            "cron_interval": 1,
            "cron_start_of_day": "00:30:00",
            "cron_interval_start": 1,
            "time_range_preset": 13,
            "delta_type": 2,
            "delta_interval": 1,
            "delta_interval_start": 1,
            "delta_start_of_day": "00:00:00",
            "datatype": "Staging",
            "creator": "jane@doe.com",
            "datastream_type_id": {{DATASTREAM_TYPE}},
            "absolute_url": "https://{{INSTANCE}}/adform/{{DATASTREAM_ID}}/adform-buyer/"
        }
    ]
}

The value in an id parameter of the response is a datastream’s ID.

Determining current datastream configuration

To determine a datastream’s current configuration, follow these steps:

1. Create a GET request to the following endpoint:

   https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
   

   Replace {{DATASTREAM_ID}} with the datastream’s ID.

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

As a result, you determined the current datastream configuration.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

Retrieving the assigned destinations

By default, a datastream detail response does not include the destinations that the datastream delivers to. To include them, add the expand query parameter with the value destinations to either datastream detail endpoint:

• GET /api/datastreams/{{DATASTREAM_ID}}/

• GET /api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/

Both endpoints return the same assigned_destinations content for the same datastream.

To retrieve a datastream’s assigned destinations, follow these steps:

1. Create a GET request to one of the following endpoints:

   https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/?expand=destinations
   

   Replace {{DATASTREAM_ID}} with the datastream’s ID.

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

As a result, the response includes an assigned_destinations array that describes the destinations assigned to the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location -g --request GET 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/?expand=destinations' \
--header 'Authorization: Bearer {{KEY}}'

Each object in the assigned_destinations array contains the following fields:

Example of the assigned_destinations array in a response:

"assigned_destinations": [
    {
        "target_id": 123,
        "destination_name": "My BigQuery",
        "destination_type": "bigquerytarget",
        "enabled": true,
        "destination_settings": {
            "table_name": "analytics.spend",
            "write_disposition": "WRITE_TRUNCATE",
            "partition_by_date": true,
            "partition_column_name": "date",
            "expires_in": null,
            "overwrite_key_columns": false
        }
    }
]

The destination_settings object contains a curated set of settings that depends on the destination type:

Determining datastream configuration options

To determine the options you can configure for a datastream, follow these steps:

1. Create an OPTIONS request to the following endpoint: Replace {{DATASTREAM_ID}} with the datastream’s ID.

   https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
   

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

As a result, you obtained the list of options you can configure for the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request OPTIONS 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'

In the response, search for specific parameters and objects to edit your datastream configuration. If a specific object includes the parameter read_only with the value true, you cannot edit it. "read_only": false means that you can edit a particular object. Fields that are auth-specific (such as accounts, campaigns, or advertisers) include "is_auth_scoped": true, indicating that the available options depend on the authorization used.

Configuring a datastream

To configure a datastream, follow these steps:

1. Create a PATCH request to the following endpoint:

   https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
   

   Replace {{DATASTREAM_ID}} with the datastream’s ID.

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. In the HTTP request header, include the parameter Content-Type with value application/json.

4. In the HTTP request body, include the names of the datastream configuration options as parameters. Specify a new value for each option.

5. Send the request.

As a result, you configured the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{KEY}}' \
--data-raw '{"name": "NEW_VALUE"}'

Listing options for filters

When configuring filters in a datastream, you can only choose from a limited set of options. For example, in the Advertiser filter, you can only choose from a specific list of advertisers. The list is populated by advertisers about which the authorization you use to connect to the data source is authorized to collect data.

To obtain the list of options for a filter, given a datastream type and an authorization, follow these steps:

1. Determine the ID of the authorization you use to connect to the data source.

2. Determine the Datastream Type’s ID.

3. Determine the Datastream Type options.

4. In the response, search for the filter you want to configure. Fields that are auth-specific include "is_auth_scoped": true. For example, the result may contain the following information about the advertisers filter:

   "advertisers": {
       "type": "multiple choice",
       "required": false,
       "read_only": false,
       "label": "Advertisers",
       "is_auth_scoped": true,
       "help_text": "If left blank, data for all advertisers available will be retrieved.",
       "lookup": {
           "url": "/adform/adformdatastream/advertisers/",
           "arguments": [
               "auth"
           ]
       }
   },
   

   Note the value of the url parameter in the response.

5. Create a GET request to the following endpoint:

   https://{{INSTANCE}}/URL_FOUND_IN_RESPONSE_OF_PREVIOUS_STEP/?auth=CONNECTION_ID
   

   Replace URL_FOUND_IN_RESPONSE_OF_PREVIOUS_STEP with the value of the url parameter from the previous step, and replace CONNECTION_ID with the ID of the authorization you use to connect to the data source.

6. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

7. Send the request.

As a result, you obtain the list of options for the filters, given this datastream type and authorization.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request GET 'https://{{INSTANCE}}/adform/adformdatastream/advertisers/?auth=123' \
--header 'Authorization: Bearer {{KEY}}'

Example of JSON response:

{
    "err": "nil",
    "results": [
        {
            "id": 123,
            "text": "Advertiser1"
        },
        {
            "id": 124,
            "text": "Advertiser2"
        }
    ]
}

Configuring local data retention

Adverity lets you control how long collected data is retained locally. You can set three retention behaviour flags and a retention policy on any existing datastream using the Management API.

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"remove_if_empty": true, "skip_empty_rows": true, "prune_instantly": true}'

{
    "retention_type": 3,
    "retention_number": 30
}

curl --location --request PATCH 'https://{{INSTANCE}}/api/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}' \
--header 'Content-Type: application/json' \
--data-raw '{"retention_type": 3, "retention_number": 30}'

Deleting a datastream

To delete a datastream, follow these steps:

1. Create a DELETE request to the following endpoint:

   https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/
   

   • Replace {{DATASTREAM_TYPE}} with the datastream type’s ID. For more information on finding a datastream type’s ID, see Determining a Datastream Type’s ID.

   • Replace {{DATASTREAM_ID}} with the datastream’s ID.

2. In the HTTP request header, include the parameter Authorization with value Bearer {{KEY}}.

3. Send the request.

As a result, you deleted the datastream.

Import the request example as raw text to your HTTP client (such as Postman). The cURL request example is the following:

curl --location --request DELETE 'https://{{INSTANCE}}/api/datastream-types/{{DATASTREAM_TYPE}}/datastreams/{{DATASTREAM_ID}}/' \
--header 'Authorization: Bearer {{KEY}}'