Are You Using Big Data, and Does It Matter?

  |  June 4, 2014   |  Comments

The majority of email marketers and marketing problems simply don't require a big data solution. And that's a good thing.

I was on a panel recently where I claimed that for 99 percent of email marketers, big data is irrelevant. I got some hefty pushback, but I stand by the claim. To clarify, let's start by talking about what big data is, and perhaps more importantly is not.

According to Wikipedia, "Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications." And the reason there's so much buzz around big data is the things that people are aiming to do with it. Specifically: crowdsourcing, data fusion and integration, genetic algorithms, machine learning, natural language processing, time series analysis, and visualization.

As a rule of thumb you can say that if you're working on it with Excel, it's small data. If you need a SQL database system (or systems) it's medium data and if you need a team running a no-SQL, cloud-based cluster running MongoDB, Hadoop, Splunk, or some MapReduce framework, it's big data.

From these definitions it should become immediately apparent that the overwhelming majority of email marketers and marketing problems simply don't require a big data solution. And that's a good thing. Big data for big data's sake has the potential to be an enormous boondoggle for many organizations.

curly-fryI'm not so jaded as to dismiss big data entirely. As the curly fry correlation shows, we can perform some amazing predictions from entirely unexpected data. (The curly fry correlation being that liking curly fries on Facebook is extremely predictive of high intelligence - Jennifer Golbeck's hugely enlightening TED talk explains why.) This type of analysis can be very valuable, but it's only possible if you have both the data and the ability to properly mine it, and very few do.

This I think is where things have fallen down for big data. Oftentimes what isn't big data is sold as such because buzzwords sell. Then, as is common with new buzzwords, there have been unrealistic expectations. We expect the curly fry correlation to tell us something deep about curly fries or about intelligence and when it doesn't it leaves us unsatisfied and disillusioned. The backlash against big data has reached the point where many dismiss it as snake oil. There's even a small data movement focused on data sets small enough to enable meaningful human comprehension and visualization.

Rather than going with either the hype or the backlash, I recommend a different course entirely. I believe we should aim for pragmatic data. We're here to do marketing. Forget the overblown claims and hype of the latest shiny object but don't dismiss new ideas just because they're hyped. Remember the Pareto principle (the 80/20 rule) - 80 percent of the effects come from 20 percent of the causes.

If you're building effective segment models and personas in Excel, then more power to you. You're likely achieving most of the value for a fraction of the cost of a larger system and doing so in far less time, which is a huge win in my book.

If you have a larger organization and you're pulling together your customer data from disparate sources into a single database and then using tools to analyze behaviors and identify segments and offers you don't need to justify it by calling it big data.

Finally, if you're actually mining petabytes of data to discern deep correlations and thereby accurately predict consumer behavior, you're one of the few. I'm impressed, slightly scared of what you'll learn, and suspect your name is either Larry or Jeff. No doubt you're driving incremental value and that's great.

In all these, be pragmatic. Email, more than most disciplines, suffers from a lack of resources. Focus on doing what works and doing so as efficiently as possible - as quickly as you can with the minimum of resources and cost. Do this and you'll achieve far more with far less than anyone expects and then no one will care how big your data is.

Images via Shutterstock.

ClickZ Live New York Want to learn more?
Attend ClickZ Live New York March 30 - April 1. With over 15 years' experience delivering industry-leading events, ClickZ Live brings together over 60 expert speakers to offer an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!


Derek Harding

Derek is the managing director of J-Labs, Javelin Marketing Group's technology skunkworks, a role that draws on his 20 years of experience and leadership in the fields of marketing and technology. A British expatriate based in Seattle, Washington, Derek is perhaps better known as the founder and technologist behind Innovyx, one of the first email service providers later acquired by the Omnicom Group. An industry veteran and thought-leader, Derek is a regular expert author, contributor, conference speaker, and takes an active role in a number of industry and trade groups.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!




Featured White Papers

A Buyer's Guide to Affiliate Management Software

A Buyer's Guide to Affiliate Management Software
Manage your performance marketing with the right solution. Choose a platform that will mutually empower advertisers and media partners!

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.


    • Website Optimizer - SEO, CRO, Analytics
      Website Optimizer - SEO, CRO, Analytics (Marcel Digital) - ChicagoMarcel Digital, an award winning interactive marketing agency established in 2003...
    • Director of Marketing
      Director of Marketing (Patron Technology) - New YorkDirector of Marketing We are seeking a Director of Marketing to manage and build our marketing...
    • Senior Interactive Producer
      Senior Interactive Producer (Ready Set Rocket) - New YorkWhat You'll Do As a member of our team, the Senior Producer reports directly to our...