Search Engine Bans and Penalties, or Just Your Imagination?

  |  August 28, 2006   |  Comments

Instead of keywords, keywords, keywords, think access, access, access.

After the recent Search Engine Strategies (SES) conference in San Jose, I was bombarded with requests to help get Web sites unbanned by Google and other major search engines. At first, I thought I was getting requests from former black-hat SEO (define) clients whose sites were getting their just desserts: exploit the search engines, get penalized.

After a few consultations, I realized a number of sites didn't even have search engine spam penalties. What I discovered were people with unrealistic expectations about search engine visibility; limited knowledge about the technical aspects of Web sites; and unclear concepts about how search engines work.

Search Engine Spider vs. Search Engine Index

I always have great fun at SES, not only as a speaker but also as an attendee. I often sit in the back of the room and listen to attendees' questions. They reveal how little or how much people know about the entire optimization process. If I hear the same question in multiple sessions, I know I should explain the concepts more clearly in forthcoming presentations.

The biggest misconception at this year's conference was the difference between the spidering and indexing processes. The index is a subset of the spider. Search engines access Web content through the spidering process. Then, they filter out duplicates and other bad content and create the index.

Maybe the reason SEO professionals, Web site owners, marketing professionals, and usability experts don't understand this concept is because there's so much emphasis on keywords. Before you build a Web site, you should perform keyword research. Keyword research is necessary to understand how users categorize and describe your products and services before you place site navigation and other sense-of-place labels on an advanced prototype. You can think keyword, keyword, keyword until you're blue in the face. Search engines won't see your Web pages' keywords unless you provide access to those keywords.

A Web site can be spidered and not included in a search engine index.

Instead of keywords, keywords, keywords, think access, access, access. Many Web site owners think their sites have been banned in the search engines when in reality they suddenly provided search engines with limited or no access to their sites' content.

Spam Penalty or Not?

Before assuming a site has been banned from or penalized by a search engine, look for technical reasons the site may not be included in the index.

Look at the robots exclusion protocol, both the text file (if used) and the robots exclusion meta tags. Was your site's content management system recently upgraded? Were the 301 redirects (define) properly implemented? Is your server delivering the appropriate HTTP status codes?

The 301 redirect is like a change-of-address card for a URL. It means a Web page address has permanently moved to a new address. If search engines have limited or no access to the new URLs, Web pages won't appear in search results.

Additionally, if 301 redirects aren't properly implemented, they can really mess up a site's link development. I recently had to go through a painstaking explanation (and some troubleshooting) of improperly formatted 301s for a client who assumed his site was banned by Google.

Ouch! Bad flashback. Excuse me for a moment while I take an aspirin.

Another common technical issue is a change in a page's linkage properties. Link farms are a form of search engine spam. A Web site owner has no control over how other sites link to his site, but he has complete control over how he links to other sites. Therefore, search engines rarely penalize a site for how other sites link to yours, with the exception of link farms. Additionally, by modifying all outbound links on a page to have a nofollow attribute, site owners change the hub or authority score of that page. That can have a negative impact on search engine visibility.

After all technical reasons for not being spidered or indexed are addressed, there's the genuine possibility a site is penalized or banned. Helping sites get unbanned is no fun task. It often takes months to undue all the damage created by black-hat SEO professionals.

I actually encountered a Web site banned for its affiliates' actions. The corporate site wasn't actually part of the link farm, but many of its affiliate sites participated in link farms to achieve instant link popularity. As a result, Google banned both the affiliate sites and the corporate site. Get a good handle on your affiliate SEO plan before embarking on an SEO campaign.

Ouch! Another bad flashback. Excuse me again while I take another aspirin.

Conclusion

First, look for technical reasons why Web pages aren't being spidered and indexed. Then, if the site has no technical issues, try to find items on a site that may violate the terms and conditions set forth by the commercial Web search engines. Sometimes, the simplest solution is turning to the black-hat SEO firm you hired and saying, "You're fired!"

All too often, the main reason people believe their sites have been banned or penalized is unrealistic expectations. All search engine positioning fluctuates. It doesn't matter whether pages of your site had top-10 positions for the past five years. No site is entitled to top search engine positions all the time.

New sites are added to the Web every day. Sometimes, these sites have better content and better site usability than your site. Search engine spam pages slip through the cracks, too. And search engines certainly don't always have effective algorithms. Keyword research, Web site development, usability, and search optimization are ongoing processes. By staying on top of all these processes, your site should receive consistent, qualified traffic over time from search engines and other reputable resources.

Want more search information? ClickZ SEM Archives contain all our search columns, organized by topic.

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Sept 5 to take advantage of Super Saver Rates!

ABOUT THE AUTHOR

Shari Thurow

Shari Thurow is the founder and SEO director at Omni Marketing Interactive, a full-service search engine marketing, Web, and graphic design firm. Acknowledged as a leading expert on search engine friendly Web sites worldwide, she is the author of the top-selling marketing book, "Search Engine Visibility," published through Peachpit Press. Shari's areas of expertise include site design, search engine optimization, and usability.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Search newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.

Resources

Jobs

    • Senior Director US Agency Ad Sales
      Senior Director US Agency Ad Sales (Expedia, Inc.) - ChicagoJob Title:  Senior Director US Agency Ad Sales   Position Overview: The Senior...
    • Senior Director US Agency Ad Sales
      Senior Director US Agency Ad Sales (Expedia, Inc.) - New YorkPosition Overview: The Senior Director US Agency Ad Sales is responsible for managing...
    • Digital Marketing Analyst
      Digital Marketing Analyst (GovLoop) - Washington D.C.Are you passionate about audience acquisition? Love effective copy and amazingly effective...