Mythbusters: Debunking the Biggest Data Lies
Big data may be the new black, but it’s also a big lie.
Big data may be the new black, but it’s also a big lie.
Harvard Business Review calls it a “management revolution.” Palantir, a start-up that used big data to help the U.S. government track down Osama Bin Laden, is now one of the hottest companies in Silicon Valley, valued at more than $9 billion, and is quickly becoming the new target for smart college grads for whom Google, Facebook, and Twitter are no longer “cool.”
To look at big data (using big data), Google shows that search volume for “big data” follows a nice hockey-stick trajectory envied by many start-ups.
It’s pretty clear – big data is big business. Everyone wishes their search terms had the trajectory that big data has. But despite all the hype, the reality is that no one actually really understands how to leverage big data.
The question is what is all the confusion about and how do we integrate big data into a manageable part of our business?
One of the biggest misunderstandings about big data is that people think it’s complicated. To some extent, this is true because working with big data involves working with technologies such as SQL databases, Hadoop, and Apache Hive.
For many without a technical background this can be quite intimidating. However, there’s another deeper reason why so much misunderstanding exists within the big data space, something I would like to call the “Data-Industrial Complex.” Just like the “Military-Industrial Complex” or the “TV-Industrial Complex,” there exists a big ecosystem of intertwined companies and stakeholders whose profits and careers depend on this artificially created smokescreen of confusion.
The usual suspects here are big data technology vendors such as IBM, Oracle, SAS, and even Amazon. Also involved are management consulting firms and digital ad agencies that peddle these technologies to their clients under the guise of helping them obtain deeper customer insights and improve business and marketing performance.
For example, at Moda Operandi, one of the previous companies I worked at, we were introduced to a big data platform (whose name I shall not name) through one of our investors. For several months, we tried adopting the product but quickly resorted back to the old days of manual SQL queries because the platform was too clunky and we were able to do more in less time using SQL to pull from our data warehouse.
No One Ever Got Fired for Buying IBM
This supplier-side bias is compounded by a consumer-side that’s often clueless about big data. C-level executives in Fortune 500 companies insist on purchasing the latest big data platforms and technologies in order to ensure their “competitive advantage.”
But we’ve found when working with our clients we often use a combination of homegrown transactional databases and third-party tools such as Google Analytics and Mixpanel to collect clickstream, behavioral, and transactional data. This allows us to increase conversion rates and average order values through onsite and email personalization.
We can incorporate our existing downstream fulfilment and delivery data and gain even more insight by consolidating all data in a data warehouse, often just a simple SQL database.
The reason why people still go for the fanciest platforms is because of fear – “a fancy tool just gives the second-rater one more pillar to hide behind,” says Hugh MacLeod, blogger, cartoonist, and best-selling author.
In his bestseller Zero to One: Notes on Start-Ups, or How to Build the Future, Peter Thiel, the billionaire venture capitalist who founded PayPal and Palantir, says, “Today’s companies have an insatiable appetite for data, mistakenly believing that more data always creates more value. But big data is often dumb data.”
The Reality Is That Most Companies Don’t Need That Much Data
The majority of us are not in the noble business of finding a cure for cancer or tracking down terrorists, but rather selling products. And we don’t need to crunch that much information to achieve our objectives.
The reason why people in (mostly big) companies still end up obsessing about more and more data is once again because they’re afraid. Afraid of making decisions based on less than perfect data, afraid of trusting their experience, afraid of taking responsibility.
Every one of us has probably experienced meetings in which one person would volunteer for devil’s advocate and point out the lack of perfect data. Projects would be delayed, profits would be lost, and careers would be ruined.
Data is only as valuable as the actions that derive from that data. For example, one of our e-commerce clients is obsessed with collecting style/persona-preferences such as “natural,” “sexy,” and “classy” to create the perfect newsletter, whereas the low-hanging fruit was to personalize based on readily available data such as gender and age.
Again, you don’t need that much data to take action.
Moving Forward: Just Enough Data
A better approach to the picky client above would be to start personalization based on data that’s already available such as demographics data while preparing infrastructure for advanced clickstream and purchase history-based personalization. Because any data that’s not actionable is useless.
Most companies are sitting on enough data but still fail to take any action. The solution requires a fundamental paradigm shift from “big data” to “just enough data.”
This is in line with the lean movement that encourages companies and employees to take an MVP (minimum viable product) approach towards building businesses and products. Big data is long overdue for an MVP revolution.
Remember, action is greater than analysis.
A Few Parting Words
Big data is here to stay as evidenced in the success of companies like Palantir and Amazon. However, people need to stop drowning in the data Kool-Aid and start focusing on getting real value out of it.
Adopting a “just enough data” mindset will focus your organization on getting actionable insights from data. And remember: this can only be done by humans, not technology.
*Homepage image via Shutterstock.