The Anatomy of a Crawler Friendly Web Page

  |  April 6, 2009   |  Comments

A reference guide to optimizing your Web pages for search engines.

On-page optimization may not have the rank-boosting power it had in the '90s. But it's the bedrock of a solid SEO (define) campaign.

Many companies use a content management system (CMS) to deliver content to their Web site. And many systems are inherently search engine crawler unfriendly. So I asked the team of experts I work with to help me create a reference guide to the anatomy of a crawler friendly Web page.

What Should Go in the Head of the Document?

  • The title is the most important tag. Use it. Make it unique to each page.

  • Meta description is the second most important heading tag. Use it. Make it unique to each page. This plus the title alone could keep pages from being considered duplicate content. The meta description probably won't improve your rankings, but it could improve CTRs (define).

  • Specify the document type (first line in the page). You probably want the following, but your developer will know for sure: .

  • Drop in a content type tag. Put it before the title tag. You probably want .

  • Every other tag is of little or no value. Author, category, and other such tags inserted by some CMS systems are ignored.

What Should Go in the Body of the Document?

  • Text as text. Use Flash and AJAX (define) for non-text elements and/or interactive elements when CSS (define) won't suffice.

  • Use images related to text on the page and not just filler. Alt tags, file names, and a caption will improve the chances of being included in image results. Use a high-resolution file or link to one. For alt tags, otherwise known as an alt (alternative) attribute, place a text description of an image so that visually impaired surfers with text readers know what the content is. Search engines can also pick up this text and the keywords in it.

  • Avoid nested tables that can result in content "appearing" differently to search engines than what you expect. Getting rid of tables makes for a lighter page, too.

  • Include related links to other articles on your site. Having them in the body will help keep them from being classified as just navigation and devalued.

  • Use images for a lengthy boilerplate or disclaimers.

What Order Should HTML Tags Be In?

  • Start your content with a heading, H1.

  • Use just one H1, and then H2 or H3 as needed. Maintain a proper hierarchy.

  • Don't use heading tags in your masthead or navigation. Keep them for the main content.

  • All content should be in between the body tags. Everything else might be ignored.

  • Watch out for tags that aren't closed. This could inadvertently hide content from search engines.

Should You Use Country/Language Tags?

  • Country and language tags aren't necessary. Search engines do a decent job, in particular, with language.

  • Want to give Google some more info? Set the geo-targeting option in Webmaster Tools.

Is a Canonical Tag Necessary?

  • Make a canonical tag part of the template so it's set automatically. (A canonical tag lets search engines know which is the primary version of a page, as in the version with www as opposed to the version without www in front. This helps to avoid indexing duplicate content.) Then you'll have less to worry about with other teams using tracking parameters on banners and other ads creating duplicated pages.

  • This is particularly useful if you've got an affiliate program. It helps consolidate all of the inbound links.

What About Page Weight/Size?

  • Page size for users and page size for search engines are different. Search engines focus on code and content. They'll eventually grab everything as is evidenced by large PDFs being indexed.

  • A slow-loading movie likely won't impact search engines, but it may cause users to bounce if it's too slow. Don't send the wrong signal to search engines by having your user's bounce and do another search.

  • Search engines will crawl more than 100 links on a page, but then you've probably got a usability issue. Categorize the links and create new pages. Help your users zero in on what they're interested in.

  • Stuffing your footer with links isn't as effective as it once was.

How Much Text?

  • A good target is 250 to 300 words. More is fine especially from a user's perspective.

  • Split articles if there are distinct topic areas to enable targeting multiple keywords. Be aware that clicking annoys users, so find a good balance.

  • Use distinct URLs for each page within a series. For instance, don't show and hide different pages using CSS.

  • Link to each page in a series using keyword rich links, not Page 1, 2, 3, or next and previous.

What About CSS Formatting?

  • CSS allows you to position text and images on a page for the visitor to see in any order you wish. So, even if the code -- which the visitor can't see -- has an item at the bottom of the page, it can still appear at the top on the visible page.

  • If you can separate CSS from the HTML markup, you'll have an easier time with maintenance.

  • Avoid tricks that involve the use of negative placement even if you're not trying to trick the search engines.

  • Be aware that if you "hide" content such as a navigation menu, search engines will still see the content and follow the links.

Of course, this isn't an exhaustive list. You may have tracking code for your analytics package in the body and other additions. But if you're new to the game, it should help you get off the ground.

ClickZ Live Toronto On the heels of a fantastic event in New York City, ClickZ Live is taking the fun and learning to Toronto, June 23-25. With over 15 years' experience delivering industry-leading events, ClickZ Live offers an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!

ClickZ Live San Francisco Want to learn more? Join us at ClickZ Live San Francisco, Aug 10-12!
Educating marketers for over 15 years, ClickZ Live brings together industry thought leaders from the largest brands and agencies to deliver the most advanced, educational digital marketing agenda. Register today and save $500!


Mike Grehan

Mike Grehan is currently chief marketing officer and managing director at Acronym, where he is responsible for directing thought leadership programs and cross-platform marketing initiatives, as well as developing new, innovative content marketing campaigns.

Prior to joining Acronym, Grehan was group publishing director at Incisive Media, publisher of Search Engine Watch and ClickZ, and producer of the SES international conference series. Previously, he worked as a search marketing consultant with a number of international agencies handling global clients such as SAP and Motorola. Recognized as a leading search marketing expert, Grehan came online in 1995 and is the author of numerous books and white papers on the subject and is currently in the process of writing his new book From Search to Social: Marketing to the Connected Consumer to be published by Wiley later in 2014.

In March 2010 he was elected to SEMPO's board of directors and after a year as vice president he then served two years as president and is now the current chairman.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Search newsletter delivered to you. Subscribe today!



Featured White Papers

Gartner Magic Quadrant for Digital Commerce

Gartner Magic Quadrant for Digital Commerce
This Magic Quadrant examines leading digital commerce platforms that enable organizations to build digital commerce sites. These commerce platforms facilitate purchasing transactions over the Web, and support the creation and continuing development of an online relationship with a consumer.

Paid Search in the Mobile Era

Paid Search in the Mobile Era
Google reports that paid search ads are currently driving 40+ million calls per month. Cost per click is increasing, paid search budgets are growing, and mobile continues to dominate. It's time to revamp old search strategies, reimagine stale best practices, and add new layers data to your analytics.




    • GREAT Campaign Project Coordinator
      GREAT Campaign Project Coordinator (British Consulate-General, New York) - New YorkThe GREAT Britain Campaign is seeking an energetic and creative...
    • Paid Search Senior Account Manager
      Paid Search Senior Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring a strategic Paid Search Senior Account Manager...
    • Paid Search Account Manager
      Paid Search Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring an experienced Paid Search Account Manager to...