Data-driven teams can’t rely on manual integration processes — they take too long to configure and are error-prone and hard to maintain. But automating data integration tasks to create a single view of your data poses its own challenges.
Not least of these challenges is that the platforms you’re pulling data from are constantly evolving, and that means the data you get from them and the way you get it might not always look the same.
When this happens, the systems automating your data integration can break, leaving marketers and data teams in the dark. So, it’s really important to understand how to maintain the connectors that are bringing together data from all these different platforms’ APIs.
This blog will look at what it takes to keep your connectors up to date behind the scenes so that you can smoothly automate a single view of your data.
What is an API connector?
An API connector is the mechanism through which information can be passed between two applications with APIs.
While APIs (Application Programming Interfaces) create the infrastructure that allows different applications to pass information between one another, until API connectors enter the mix, fetching data via an API for your marketing reporting purposes can not easily be done without a level of engineering. Connectors are the mechanisms that enable communication between two applications, allowing marketers to easily access and retrieve the information they need.
You can read more on the difference between connectors Vs. APIs here.
Why is maintaining API connectors important?
Wherever you end up sending your data and reports, the end users will value having a data architecture that doesn’t break! Maintaining your API connectors ensures a continuous flow of information.
So, for example, if you’re in the middle of a marketing campaign, but suddenly don’t know how the campaign is performing, this is very bad! The process might look something like this:
- A platform updates its API and your connector breaks
- Interruptions occur in your data flows
- Reporting and datasets aren’t delivered on time
- There is no data to drive timely decisions and optimize activities
- Opportunities are missed and risks are detected too late
- Budget is wasted and campaign goals aren't met
- Department fails to prove value to the C-suite
So, without proper API connector maintenance, organizations could end up wasting their budget, and struggling to explain the value of their activities to the C-suite.
How do you maintain API connectors?
We’ve built hundreds of API connectors that have to be regularly maintained, and it’s no simple task. Over the years we’ve picked up a few tips that we can share with those of you looking to build a single source of truth by building your own API connectors - or anyone interested in what goes into connector maintenance.
1. Identifying API updates
Step one is to understand what is being updated within the APIs and when. Sounds simple, right? But there’s actually quite a bit that goes into this. There are four key pieces of information you need before you can start planning updates to your API connectors.
2. When are API updates happening?
For each API connector you’ve built, you need to know the API update release cycle so that you can plan updates for your connectors ahead of time.
This isn’t quite as simple as it sounds. Every API is different – some don’t get updated for years while other APIs are updated much more regularly, and we need to check monthly, e.g. Criteo Marketing API has a three-month release cycle, with each version being available for 1 year. Some platforms update their API every month and run various versions of their APIs in parallel.
So, to keep track of API updates, you’ll need a calendar overview of all the platforms and their API update cycles. This will let you see ahead of time when to check the API. The timing is just one part of this though.
3. Where are details about API updates being posted?
Different vendors have different ways of communicating updates. Google uses an RSS feed, but other platforms might send emails or post updates on their website. So, as part of the process to check connectors regularly, you’ll need to know where to look.
If you have partnerships with your data sources then lean on these to get direct access to the information much more easily. Having partnerships also helps to clarify open questions regarding API updates quicker and you will have a direct line of communication with the vendor.
4. What are the API updates?
Having an idea of your API update cycles gives you a headstart. But you still need to do your research in each instance of an update. This involves going through the API update release notes, and the updated API documentation, so that you can understand exactly what changes are happening and when they will come into effect. It is essential to understand what impact an API update will have on your connector. Will all metrics and dimensions be named the same after the API update? Will some report types be removed? Or will everything stay the same and no changes to your reporting data is expected?
How should API updates be prioritized?
You’ll need to understand when you have to act quickly. So, once you know what’s happening and when, you need to prioritize these updates to your connector.
Executing API connector updates
So you’ve got all the information you need to update your connectors now - but how do you actually put this all into motion? There are so many moving parts in a data flow that even small updates can cause big disruptions.
1. Determine the impact of the updates:
If a field is being removed for example, are there any employees who use this field? What is the impact for them? Is there a replacement for this field?
Connector maintenance is hugely impacted by how granular you want your API to go. If you need your reports to go into a lot of detail, then maintenance can be very time-consuming.
2. Write your updated code:
Make sure that you have your API connector updates ready to go when the last version is due to be sunsetted.
3. Inform employees who must take action:
Plenty of changes can be automated. However, you may need input from the end users on occasion.
There may be some action required from your colleagues, for example, Google AdWords – completely sunsetted their API, so we had to migrate everyone’s old data and mimic these with the new API.
Try and give your team as much time as possible with clear steps on what needs to be done.
4. Create a backup plan:
If your updated connector doesn’t work, you’ll need to be able to transfer data if needed. Stuff might be missing - you need to be able to migrate and integrate.
5. Test the connector update:
Create a dedicated QA team to test every API connector update so you can be sure that everything runs as seamlessly as it should.
Conclusion
Keeping API connectors up to date might not be the most glamorous task - but without these updates running seamlessly in the background, organizations can face significant disruptions. So it’s worth getting a process in place if you’re going to be integrating data in-house. Keeping a calendar of API updates and where to find them can help you keep track of what needs to be updated, and understanding the impacts of these updates, alongside rigorous testing will ensure that any updates are as seamless as possible.
