Why is data integration still such a challenge?

Even in an industry as dynamic as ours, it’s always interesting to see change happen in real time.

What’s surprising is that even this far into the data-driven era, the changes don’t come easy. For example, we know we’re going to keep seeing different data types from different sources—and that it’s good for business. With modern marketing, it makes a world of difference: every campaign can be improved with greater knowledge of customer tastes, needs and behaviors.

So why is it still such a problem?

Well, the 2018 Data Connectivity Annual Report from Progress paints a less-than-pretty picture of the current landscape: Turns out a lot of the companies surveyed face serious challenges with data integration, and for many reasons. Accurate and timely intelligence that can effectively guide business initiatives will only come through a comprehensive and unified view of all appropriate data, and that in turn requires collating and cleaning different data sources, APIs that come and go, multi-sourced environments, and even brand new data types that disrupt existing processes.

All this, as the report makes clear, causes numerous headaches. Here’s a sampling:

  • 47% are essentially battling the glut of APIs;
  • 36% worry about data inconsistency, incompleteness and ambiguity;
  • 32% have issues with velocity (everything from batch to streaming); and
  • 31% fear variety (structured, semi-structured, and so on).

I’m willing to bet that no matter how much technology innovation we all rave about as an industry, 99% of the companies we work with at any given time are running into these obstacles right now.

And if it all sounds familiar … it should.

We’ve seen these kinds of problems before, such as with the first waves of email campaigns. It was a massive problem because there were inevitably different types of email in sources, format and more, and they needed to be handled differently as a result. It’s the digital universe’s incarnation of natural evolution: New types and formats get replaced by newer types and formats, and it’s up to companies in this space to move with the times.

We need to learn from the past: The incoming tidal wave of data isn’t going to recede anytime soon, and it’s up to the companies involved to implement a different approach with a different kind of infrastructure.

First, let’s accept that if data integration is a big problem now, it will become a bigger problem in the near future. APIs will become more specific and more diverse, we’re going to see more devices in more form factors with more apps generating more semi-structured or even unstructured data, and we’re still in the dawn of the IoT era, and that discipline alone could explode many existing data environments. Even some of the potential solutions will bring more problems. For example, area-specific silos may solve some problems, but they’ll add to the integration complexity. So if integration is already a headache, it will become a nightmare soon.

Or maybe not.

If it’s technology innovation that brings us these logistical nightmares (new data sources! different data types! more APIs!), it also delivers realistic solutions.

There are now technologies available that allow companies to change their entire approach: Instead of creating multiple data silos to manage different data streams, they can develop a centralized data ecosystem in which all data—online and offline—resides on the same platform. In this arrangement, marketers can access data from all touchpoints.

This a radical departure from the approaches of the past, and that’s really the point. Technologies don’t function in a vacuum—they meet pressing business needs and offer tantalizing business opportunities. To truly optimize the data in hand to develop actionable business intelligencer, we must first take on the challenge of data integration—and that’s a critical business issue, not a technology one-off.

Too many organizations still function in an environment where much of the data is confined to separate silos—and a lot more data remains out of reach. Remember, many companies don’t own or control significant volumes of data that’s rightfully theirs, in part because different third parties are deeply embedded in the process of data gathering. These vendors attach their tags to just about all market-facing content, and pass along only segmented slivers to the brands paying for the campaign. There can even be reluctance to try to bring this data home—after all, look at the trouble everyone’s having with the data they’ve already got.

Again, I believe companies are on course to wake up from the integration nightmares they still face. The technologies needed to optimize integration from multiple and complex data streams are available. It only takes a change in approach and operating philosophy—and sometimes, that’s a bigger challenge. Let’s hope the eventual reality of 2019 looks very different from the forecast we see now.

 

A relationship builder, Marc Sabatini has more than 20 years of experience consulting on technology and data solutions in direct marketing, CRM and omnichannel digital marketing. He is currently the chief revenue officer at Aqfer, and has extensive enterprise expertise and has worked with solution providers like Epsilon, Donnelly Marketing and InfoUSA, as well as founding a successful startup consultancy, Advantage Insight, which was acquired by Redi-Data.

 

If you enjoyed this article, sign up for SmartBrief’s free email from the ANA Data & Analytics SmartBrief, among SmartBrief's more than 200 industry-focused newsletters.