Adventures of a Big Site's Migration

So, you’ve been online for 10 years. You have a huge Web site with over 80,000 pages. Google loves you and has blessed you with tons of traffic. You’re very comfortable with your online presence. So the last thing you’d ever think about is doing anything risky like switching to a new domain. In fact switching to a new domain, a new host and in a different country. Now that, would be sheer madness wouldn’t it?

As a long-timer in this business, I’ve seen the look of terror on the faces of Webmasters who simply need to switch to a new host, let alone change everything that possibly could be changed! The fear of losing search-referred traffic is a real one.

When I discovered that my friends at e-Consultancy, the U.K.’s largest digital marketing site, was about to do that, I was very keen to follow its progress.

Last week I spoke with e-Consultancy CEO Ashley Friedlein, asking him if he had lost his mind. Here is what he had to say about the company’s move.

Mike Grehan: Why on earth would anyone sitting so comfortably want to make such dramatic changes?

Ashley Friedlein: We had to change the site so we could improve the user experience. We wanted to upgrade our technology, plus we’re expanding into the U.S. so we thought it would make sense to have the site hosted there.

The directory structure and the URLs changed as part of the information architecture change.

The domain change was more of a brand thing. It was quite a small change on the surface, in as much as we just dropped the hyphen from e-consultancy. But of course, it literally is a different domain and then we also dropped the www to make the URL shorter and partly because we thought that this was becoming a bit of fashion À la the Twitters of this world.

We ended up changing the hosting company, the country, and the domain name — effectively everything you could change, we did!

From a user perspective, it’s not a big change. But from a Google perspective it was a completely new entity.

MG: Knowing just how risky the whole thing would be, how did you set priorities and then implement everything?

AF:There were kind of only two things we focused on.

One, quite simplistically, was to make sure we had the new site verified with Google’s Webmaster Tools. Not that would help us with rankings or anything. But because it would mean that we could see whether we were being crawled extensively and whether or not the inbound links were being recognized.

Just making sure that the transition from the old domain to the new was happening and trying to get some insights into what was going on in Google’s brain, as it were. Important, but fairly simple to do.

A much more complex exercise was making sure we got the 301 permanently moved redirects accurately in place. We wanted to ensure that any corresponding page on the new site that existed on the old site was the same page, same content at the new domain.

We had about 80,000 pages, so this is not something you can do manually. Where that content existed in a database meant it was relatively easy to write a script to do the URL mapping. One other very useful thing was that the old site was well documented in Google’s Webmaster Tools. So we were able to download all of the external links that Google was seeing pointing to the old site. And we went through those to make sure we’d redirected those.

In some cases we had links pointing to pages which weren’t necessarily visible in the navigation on the old site. And there was some stuff that we did do manually. Flat HTML content outside of the database such as the “about us” pages and other stuff we had to manually redirect.

Basically the task was to make sure that every single page on the old site was redirected to its exact, or nearest equivalent, on the new site. Or, if it had genuinely disappeared, then we let it go to a custom 404 page, given that we ultimately wanted it de-indexed as it didn’t really exist anymore.

According to Webmaster Tools at Google, we had nearly 70,000 inbound links, which we had accumulated over time. But there was no way we were going to try and get everyone to update those links, it just isn’t realistic. I mean, we are making a bit of an effort to update the key links we recognize as being the most important.

We just wanted to make sure that all of the links to pages on the old site cleanly redirected to the equivalent page on the new site. And that seems to be working well as Google is now showing something like 90,000 links to the new site. So it’s clear that Google has figured this out is now treating links to the old site as links to the new site.”

MG: It’s all very confidence-building stuff. But what was the outcome. Did it work according to plan?

AF: Well it’s a yes and no thing. It would appear that search engines, certainly Google and Yahoo, get up to speed pretty quickly with the indexing of a new site and de-indexing of the old. So at the page level, that kind of migration seems to have worked just fine. But much as we’ve discussed the deceiving nature of PageRank, it migrated to the new site in an odd way. The PageRank for the section pages migrated and showed up almost instantaneously. But the PageRank for the home page dropped considerably so that now, PageRank for the home page is half of that for the section pages — which is strange.

So the technical stuff went sort of according to plan. But any rankings we had have dropped very considerably and our referrals from Google are way down and the traffic we are getting is almost all brand terms. It’s about seven weeks since we made the change, so either Google is still trying to figure everything out and data is still percolating through. Maybe it’s the trust thing?

MG: So should people really be afraid to make these types of changes?

AF: Not really. I mean we made the change for genuine business and end-user reasons. We did it exactly how Google tells you to do it. So, if it doesn’t work, that’s because Google must be broken — not us. It’s Google’s problem to figure it out, not ours. We did it the way they told us.

Addendum: Ashley is charting progress at his blog. It makes very interesting SEO (define) reading. You’ll find it here.