Skip to content
Book a demo

Connect documentation

All documentation
Documentation

Scaling datastreams

On this page

This guide explains how to scale your datastreams in Adverity.

Introduction

Expand your data operations and make changes to your data collection setup at scale. Having many datastreams and changing data needs comes with repetitive tasks for multiple datastreams. In Adverity, you can perform these tasks efficiently for all affected datastreams at once.

Fetching data manually for multiple datastreams

To start a manual fetch for multiple datastreams, follow these steps:

  1. Go to the Datastreams page.

  2. Select the datastreams for which you want to start a fetch.

  3. Above the datastreams list, click Fetch now.

  4. In the pop-up window, configure the fetch parameters. For more information, see Fetching data manually.

  5. Click Fetch.

Enabling and disabling schedules for multiple datastreams

To enable or disable schedules for multiple datastreams, follow these steps:

  1. Go to the Datastreams page.

  2. Select the datastreams for which you want to enable or disable schedules.

  3. Above the datastreams list, click Schedule fetches.

  4. Configure the schedules and click Set schedule.

Enabling and disabling all destinations for multiple datastreams

To enable or disable all destinations for multiple datastreams, follow these steps:

  1. Go to the Datastreams page.

  2. Select the datastreams for which you want to enable or disable all destinations.

  3. Above the datastreams list, click Edit destination, then click Enable destinations or Disable destinations.

Premium scalability actions

Prerequisites

Only users with Administrator permissions can perform premium scalability actions.

Renaming multiple datastreams

You can rename multiple datastreams in the selected workspace or during workspace cloning. For more information, see Cloning workspaces.

To rename multiple datastreams in the selected workspace, follow these steps:

  1. Go to the Datastreams page.

  2. Select the datastreams you want to rename.

  3. Above the datastreams list, click Rename.

  4. Select and configure a renaming rule from the following options:

    • Find and replace

      To remove found matches, leave the Replacement field empty.

    • Add prefix

    • Add suffix

  5. (Optional) Click + Add another rule to create a more complex renaming rule. Multiple renaming rules are applied sequentially from top to bottom.

  6. Click Save.

Cloning multiple datastreams

You can clone multiple datastreams at once within the same workspace or to a different workspace. During multiple datastream cloning, you can:

  • re-use, clone, or replace authorizations

  • re-use or clone transformations

  • rename datastreams

Schedules, Data Mapping, datastream settings, and local data retention settings are preserved for the cloned datastreams.

For more information about other types of cloning, see:

Limitations

  • Cloning of destinations and Bundle datastreams is not supported. You can only assing the default destination in the target workspace.

  • Renaming and modifying transformations is not supported.

Cloning multiple datastreams

To clone multiple datastreams, follow these steps:

  1. Go to the Datastreams page.

  2. Select the datastreams you want to clone.

  3. Above the datastreams list, click image1 Clone.

  4. In the Workspace selection step, select the workspace to which you want to clone the selected datastreams.

  5. Click Next.

  6. In the Authorizations step, adjust the authorizations used by the selected datastreams. Click image2 Change to edit which authorization is used by a datastream.

    You can set the authorization used by each datastream to one of these options, if applicable:

    • Re-use the authorization from the parent workspace

    • Clone the original authorization

    • Use an authorization from the target workspace

    • Use an authorization from other workspaces

  7. Click Next.

  8. In the Transformations step, select one of these options:

    • Include transformations

      Select this option to clone selected datastreams including all or some assigned transformations. If the transformation belongs to the parent workspace, it will be re-used. If the transformation belongs to the source workspace and cannot be re-used in the new workspace, it will be cloned.

    • Ignore transformations

      Select this option to clone selected datastreams without any assigned transformations.

  9. (Optional) If you selected Include transformations in the previous step, configure the list of assigned transformations for each of the selected datastreams. The order of the assigned transformations is preserved.

  10. Click Next.

  11. In the Destinations step, select one of these options:

    • Add default destination

      Select this option to clone selected datastreams and assign the default destination of the target workspace to them.

    • Ignore destinations

      Select this option to clone selected datastreams without assigning the default destination.

  12. Click Next.

  13. Review the summary of the datastreams to be cloned.

  14. (Optional) To rename the cloned datastreams, click image2 Rename datastreams. For more information, see Renaming datastreams.

  15. Click Clone image3.

What’s next?

Once the datastream cloning is complete, you may need to perform the following actions to finish your setup:

Assigning a destination to multiple datastreams

Limitations

Only destinations that do not require additional configuration on the datastream level can be assigned to multiple datastreams at once. You can assign the following destinations to multiple datastreams:

  • Azure Blob

  • Azure Synapse

  • Databricks

  • Databricks (SQL)

  • File

  • File OneDrive

  • Firebolt

  • Google BigQuery

  • Google Sheets

  • LinkedIn Matched Audiences

  • Microsoft SQL Server

  • SAP Data Warehouse Cloud

  • SAP Marketing Cloud

  • Snapchat Audiences

  • Snowflake

  • SQL Database

Assigning a destination to multiple datastreams

To assign a destination to multiple datastreams, follow these steps:

  1. Go to the Datastreams page.

  2. Select the datastreams to which you want to assign a destination.

  3. Above the datastreams list, click Destination, then click Add destination.

  4. Select the destination that you want to assign and click Save.

    The destination will be assigned to selected datastreams.

  5. (Optional) Disable the destination for specific datastreams.

  6. (Optional) Load data previously collected in the selected datastreams to the newly assigned destination. Data will be loaded only when the destination is enabled.