Measurement Without Experimentation is Vanity – How To Do Big Data Better

Big data has yet to yield major new insights or spark meaningful change within many organizations. This isn’t a fault of data, but more a result of our inability to reframe its storytelling capability.

As Edward de Bono says: “Most executives, many scientists, and almost all business school graduates believe that if you analyze data, this will give you new ideas. Unfortunately, this belief is totally wrong. The mind can only see what it is prepared to see.”

The limits of perception are the limits of knowledge. The insights that big data can yield are limited by the quality of the questions an organization can ask. The culture of an organization is tied to its ability to see and perceive differently from “business as usual”.

All stories start with a language, and the language of business is data (on the face of it, at least). The problem is not that data tells stories, but that we’re complacent about hearing the same ones. We need to examine and re-design the experiments we use to interrogate the data, to start asking better questions.

Most large organizations are culturally unprepared for the opportunities that big data offers, especially in terms of their ability to re-imagine themselves and re-shape the industry they operate in. With big data, organizations have a choice: they can either discover new stories and new ways to thrive in times of change, or keep telling the same stories and re-quantifying what they already know.

Exploding some myths

1. Big data is quantitative and qualitative

Most organizations routinely privilege quantitative data over qualitative data, misunderstanding the function of each and the relationship between them.

Qualitative techniques are investigative. The data generated is typically theme and insight based. This is great for re-framing a situation, discovering the right problem to solve and generating a set of hypotheses.

Quantitative techniques are analytical. The data generated is statistical and number based. This approach is great for validation and verification – for proving and disproving hypotheses.

As Benjamin Yoskovitz (Lean Analytics) notes: “Use qualitative to discover and quantitative to prove”.

Big data is quantitative and qualitative. Once this relationship is understood, you create a hypothesis engine for your business that can scale with speed and impact – a powerful source of strategic value.

2. There’s no such thing as raw data

There’s no such thing as raw data, and no data is objective. Decisions about what to measure and why are always explicitly or implicitly made. Such decisions reflect the culture of an organization and are constrained by what is considered to be acceptable knowledge. This limits the scope of insight that big data is able to provide.

3. Data does not make decisions

Data creates a frame of reference for decisions to be made. In many organizations, that frame of reference may stifle decision making, instead of enabling it. Gathering the wrong data in the “right way” diverts attention away from real issues and opportunities. Using data as a political tool to defend the status quo is bad business strategy, it’s much better to test opinions, assumptions and closely held beliefs instead.

Stop measuring, start experimenting

Most organizations will easily build the technical competence to capture as much data as possible. The challenge will be to develop the ability to understand and interpret that data in an intelligent, meaningful and actionable way.

In order to meet this challenge and discover sustainable advantages, businesses need to adopt an experimental mindset:

a. Build an “Experiment Design” capability

Experiment design is one of the most important emerging business practices surrounding big data and analytics. In the same way that financial fluency is key to running a successful business, so is a culture of experimentation. Done well, this capability will position you as a market leader.

b. Expand the scope of big data within your organization

Big data is not just the realm of the marketing department. Expand its scope to include all parts of the organization, so new frames of reference can be created, and new stories can be told.

Using big data to run a business experimentally is not only a powerful risk mitigation strategy, it’s a profoundly sensible way of navigating a chaotic and ever-changing market.

Related reading