How can marketers make sense of the glut of insight they have at their fingertips? We ask Dixon Jones of Majestic SEO, which provides billions of bits of competitive and link data to search marketers and digital PR professionals around the world.
With our digital marketing conference, ClickZ Live New York, about to commence in just under a couple of weeks, we've reached out to some companies who'll have representatives speaking or exhibiting for some words of wisdom ahead of the event.
One of the agenda session tracks this year will focus delegates' minds on all-important "Business Intel" and how marketers can make sense of the glut of insight they have at their fingertips to help them make better decisions about where they focus their time, what works and what levers they can pull to optimize their digital marketing efforts.
In this interview, we turn to Dixon Jones from UK-based analytics firm, Majestic SEO, who are no strangers to providing billions of bits of data that search marketers and digital PR professionals around the world use for competitive analysis, link building and influencer discovery and outreach.
ClickZ (CZ): The term "big data" is thrown around a lot in our industry. Marketers have always used data to make calls on where to spend money and track results, so why the focus on "big" and why is it important?
Dixon Jones (DJ): What has changed is that we are starting to derive meaning from different - unstructured – groups of disparate data.
For example you might know all the Twitter profiles of everyone on the Internet in one data set and a list of how links to those profiles in another data set. From these you can derive how influential each Twitter user is.
If you then work out in a third data source what each person tweets about, then you have a list of the most influential person on any industry (that uses Twitter). Do this again for Facebook profiles and LinkedIn profiles and you start to get a pretty good list of all the influencers in an industry – even though that data set does not exist independently.
CZ: Why are insights from big data different to insights from "just" data?
DJ: There are a number of ways to answer that – and the combination of multiple unstructured data sets is one. But the fact that we are using the data to direct the insight, rather than using it to verify a hypothesis is (I think) a key factor.
In the example above, we end up with a list of people that influence a market sector. In the "old world order" you might say "Lady Gaga is probably influential on Twitter, can we check that?" In the new big data methodology she will be influential if she is in the list and won't be influential if she is not in the list. Whether any other list says she is or isn't becomes irrelevant, unless that other list gets fed into the unstructured data sources as well, in which case the whole list might change as a result.
CZ: Is big data relevant to small businesses or is it all about big budgets?
DJ: Both have important roles to play. Collecting and string huge amounts of data requires big budgets – but interrogating large sets of unstructured data is not, and luckily the web itself is the largest set of unstructured data on the planet.
Small businesses can go a long way into analyzing the web without having to be a custodian of the big data set. This does not always have to cost money – increasingly you can access this data through APIs, often for little money or for free. Other times you can crawl the web or scrape the data – however (and this is important), scraping data needs to take into account the rights to use the data.
There are two disciplines developing, though: data collection and data manipulation. The custody of the data is rarely well equipped to glean insight quickly.
For example, the electoral registrar is a massive data set, but the people that compile the list day-to-day have to concentrate on the data collection. Knowing that people in one ZIP code are 30 percent richer than those living in another code or 20 percent less likely to have subsidence after merging that data onto a geological map of the USA is not something the electoral registrar is likely to be very good at.
Small businesses have the agility to develop the insights, but being a custodian of the data comes with an inability to react quickly or create opportunity. The custodians create potential; businesses of all sizes can turn it into something valuable.
CZ: So what does it take to be a custodian of a big data set?
DJ: The first thing to understand is scale. Often understanding this takes a little more insight than raw numbers.
For example, Majestic crawls over 2 billion pages a day and in the process sees something like 7 billion links. Our big data set is that list of links, going back over seven years. Already that sounds a lot right?
Now consider this: we are told by Twitter that in an average day there are 500 million tweets. In our industry we think we have a better grasp of how big Twitter is. Five hundred million tweets. Sounds a lot, but it is a load less than 7 billion URLs seen in the same 24 hour period.
Then, once you consider that, consider this. Google says that the average web page is 370kb. By my reckoning, then, to see those 7 billion URLs from 2 billion crawled pages, we have to do the equivalent of trawling through about 6,500 hours of video in data every day. My math may be off, but it makes my point.
You need to understand scale at the outset. Then, when you understand the scale of the task, you hopefully realize that with that comes a strong need for responsibility. For most big data sets the biggest responsibility is one of security.
Very few data sets don't have something personal in them and if the data is anything like full, then it won't have had every person actively and willingly opting in to you giving it lock-stock to every hacker on the planet.
In Majestic's case the data is not personal, but we do have to operate responsibly in both the collection of the data – by not bringing down servers through over aggressive crawling, for example. We obey commands that websites can give us to either not crawl or crawl at a slower speed to help this and we also aim to be a very efficient crawler… but many an irresponsible data collector has inadvertently caused mayhem by being too aggressive through poor programming.
CZ: If our readers wants to dive in and capitalize on this trend, where should they start?
DJ: It depends on whether you want to capitalize on the technologies or the business opportunities. Of course they overlap.
If I was waking up tomorrow with a view to building a big data set, I would start by going on a Hadoop course. It's open source and not the only data set you are going to use, but is really is designed to help deal with "big".
If I wanted to look at the business opportunities, then there are a load of conferences and ClickZ Live has a track on "Business Intel" which is a great start. I think the challenge, though, is finding the visionary (or being the visionary) that has the ability to see the data sources, coordinate the harnessing of them all and have the political muscle in an organization to develop the business deals between the data sources.
CZ: Thanks Dixon and have a great show in NYC!
Companies who'll be speaking on the "Business Intel" track at ClickZ Live New York include Covario, Chango, iProspect, SEER Interactive, GroupM, Time Warner Cable Media, Conductor and eBay, so you'll not be bereft of intriguing insight and a whole host of inspiration if you make the journey to New York to hear what they have to say.
Registering for ClickZ Live NY is easy, just follow this link.
Melanie White is the Editor of ClickZ, where she has worked since July 2012. White has been with Incisive Media (ClickZ's parent company) since 2009 where she was Deputy Editor/ US Editor for one of the financial risk management titles, International Custody and Fund Administration (later Custody Risk).
In addition, White worked on FX Week where she was a Senior Reporter. She has also worked and contributed to a number of other titles, including The Accountant, International Accounting Bulletin, the New York Post, Independent Magazine (UK), as well as the broadcast title Sunday Live with Adam Boulton at Sky News.
2015 Holiday Email Guide
The holidays are just around the corner. Download this whitepaper to find out how to create successful holiday email campaigns that drive engagement and revenue.
Three Ways to Make Your Big Data More Valuable
Big data holds a lot of promise for marketers, but are marketers ready to make the most of it to drive better business decisions and improve ROI? This study looks at the hidden challenges modern marketers face when trying to put big data to use.
December 2, 2015
1pm ET/ 10am PT
Wednesday, December 9, 2015
5pm HKT / 5am ET