Are You Using Big Data, and Does It Matter?

  |  June 4, 2014   |  Comments

The majority of email marketers and marketing problems simply don't require a big data solution. And that's a good thing.

I was on a panel recently where I claimed that for 99 percent of email marketers, big data is irrelevant. I got some hefty pushback, but I stand by the claim. To clarify, let's start by talking about what big data is, and perhaps more importantly is not.

According to Wikipedia, "Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications." And the reason there's so much buzz around big data is the things that people are aiming to do with it. Specifically: crowdsourcing, data fusion and integration, genetic algorithms, machine learning, natural language processing, time series analysis, and visualization.

As a rule of thumb you can say that if you're working on it with Excel, it's small data. If you need a SQL database system (or systems) it's medium data and if you need a team running a no-SQL, cloud-based cluster running MongoDB, Hadoop, Splunk, or some MapReduce framework, it's big data.

From these definitions it should become immediately apparent that the overwhelming majority of email marketers and marketing problems simply don't require a big data solution. And that's a good thing. Big data for big data's sake has the potential to be an enormous boondoggle for many organizations.

curly-fryI'm not so jaded as to dismiss big data entirely. As the curly fry correlation shows, we can perform some amazing predictions from entirely unexpected data. (The curly fry correlation being that liking curly fries on Facebook is extremely predictive of high intelligence - Jennifer Golbeck's hugely enlightening TED talk explains why.) This type of analysis can be very valuable, but it's only possible if you have both the data and the ability to properly mine it, and very few do.

This I think is where things have fallen down for big data. Oftentimes what isn't big data is sold as such because buzzwords sell. Then, as is common with new buzzwords, there have been unrealistic expectations. We expect the curly fry correlation to tell us something deep about curly fries or about intelligence and when it doesn't it leaves us unsatisfied and disillusioned. The backlash against big data has reached the point where many dismiss it as snake oil. There's even a small data movement focused on data sets small enough to enable meaningful human comprehension and visualization.

Rather than going with either the hype or the backlash, I recommend a different course entirely. I believe we should aim for pragmatic data. We're here to do marketing. Forget the overblown claims and hype of the latest shiny object but don't dismiss new ideas just because they're hyped. Remember the Pareto principle (the 80/20 rule) - 80 percent of the effects come from 20 percent of the causes.

If you're building effective segment models and personas in Excel, then more power to you. You're likely achieving most of the value for a fraction of the cost of a larger system and doing so in far less time, which is a huge win in my book.

If you have a larger organization and you're pulling together your customer data from disparate sources into a single database and then using tools to analyze behaviors and identify segments and offers you don't need to justify it by calling it big data.

Finally, if you're actually mining petabytes of data to discern deep correlations and thereby accurately predict consumer behavior, you're one of the few. I'm impressed, slightly scared of what you'll learn, and suspect your name is either Larry or Jeff. No doubt you're driving incremental value and that's great.

In all these, be pragmatic. Email, more than most disciplines, suffers from a lack of resources. Focus on doing what works and doing so as efficiently as possible - as quickly as you can with the minimum of resources and cost. Do this and you'll achieve far more with far less than anyone expects and then no one will care how big your data is.

Images via Shutterstock.

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Oct 3 to take advantage of Early Bird Rates!


Derek Harding

Derek Harding is the CEO and founder of Innovyx Inc., a member of the Omnicom Group and the first e-mail service provider to be wholly owned by a full-service marketing agency. A British expatriate living in Seattle, WA, Derek is a technologist by background who has been working in online marketing on both sides of the Atlantic for the last 10 years.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!



Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising
If you're considering implementing a marketing attribution model to measure and optimize your programs, this paper is a great introduction. It also includes real-life tips from marketers who have successfully implemented attribution in their organizations.


    • Tier 1 Support Specialist
      Tier 1 Support Specialist (Agora Inc.) - BaltimoreThis position requires a highly motivated and multifaceted individual to contribute to and be...
    • Recent Grads: Customer Service Representative
      Recent Grads: Customer Service Representative (Agora Financial) - BaltimoreAgora Financial, one of the nation's largest independent publishers...
    • Managing Editor
      Managing Editor (Common Sense Publishing) - BaltimoreWE’RE HIRING: WE NEED AN AMAZING EDITOR TO POLISH WORLD-CLASS CONTENT   The Palm...