Data Integration, Part 1: Macro Integration

During the session on multichannel metrics at Search Engine Strategies (SES) in New York a few weeks ago, we had a question about integrating Web analytics data with other forms of data. What’s the best way to do this? As is often the case with Web analytics, the answer is usually, “It depends what you’re trying to do.”

One challenge digital marketers typically face is the number of different data sources they deal with at any one point in time. These might include data from:

• Ad-serving systems
• PPC (define) bid management systems
• Web analytics systems
• Affiliate management systems
• Transactional systems
• Customer management systems

The list can goes on. How do you pull all this stuff together to get a sense of what’s really going on?

I tend to think about data integration on two levels: macro integration and micro integration.

Macro integration is the process of pulling the outputs of the various data systems together into one place so you can see relationships between different data sources. Excel is often the ultimate macro data integration tool. Legions of analysts all over the country spend hours pulling together spreadsheets from multiple data sources to provide a summary of what’s going on.

This isn’t necessarily a bad thing. Pulling data together means the analyst is at least looking at them and is likely to spot any data quality problems. Problem is, 90 percent of the analyst’s time is spent on data production. Only 10 percent is spent on data analysis. The value equation is all wrong.

One approach is to try to automate the process as much as possible and get the various data sources into a single database. The data are likely to be processed or summarized rather than displayed in their original format, but at least they’re together which makes reporting out across the range of key performance indicators (KPIs) and key metrics easier. Inevitably, this requires some upfront investment to set up the data feeds and reporting system correctly. Standard databases and business-intelligence reporting tools could be used for this type of work.

At a company I once worked at, we took this approach to reporting on our key metrics: We extracted summarized visit, unique visitor, and page view data from our Web analytics system and stored it alongside other key company data in a data mart specifically designed for that purpose. The extracts were done nightly, so every morning the key site metrics were available and reported against some of our other key transactional data using a business intelligence tool (in our case, business objects).

Following the inevitable initial teething problems, the process worked quite well. Questions from management and colleagues changed from when the reports were going to be ready to what the reports were actually telling them in terms of business impact. We were able to go from spending 90 percent of an analyst’s time on data production to 90 percent of an analyst’s time on…analysis!

Obviously, this comes with a cost. Managing multiple data sources is a messy business. Significant resources may be required from your IT team, and they may or may not have the necessary experience dealing with this particular data type. An alternative route may be to outsource the process to such companies as BlackFoot Inc. that specialize in this type of data integration activity.

Next, I’ll look at micro integration — the challenges and opportunities in pulling together and integrating data at the customer level.

Till then…