Restricting crawler access

This post belongs to a series about search engine optimization (SEO) with Magnolia CMS. Today we look at restricting crawler access.

While SEO is mostly about about enticing that crawlers to your content, there are some pages you may not want them to access. These include administrative, password protected, and paid access pages.

A robots.txt file restricts crawler access on a site-wide basis. The file is typically placed in the root directory of your site. All legitimate crawlers it check fro the presence of a robots.txt file, and respect the directives it contains. Even if you do not want to restrict access, it is advisable to include a robots.txt file because the absence of one has been known to produce a 404 error, resulting in none of your pages being crawled. This can also be the case if the file is not constructed correctly. Specific crawlers can also be denied access by entering their names or IP addresses in the file.

The following example instructs all crawlers to keep out of the private and temp directories and denies access to a specified crawler.
User-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot
Disallow: /private/
Disallow: /temp/
It is also possible to exclude individual pages with a robots meta element in the page HTML header.
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
Relevant values for the robots meta element are:
  • NOINDEX prevents the page from being included in the index.
  • NOFOLLOW prevents the crawler from following any links on the page. Note that this is different from the link-level rel attribute which prevents the crawler from following an individual link. For anti-spam reasons, it is generally better to use the rel attribute with value NOFOLLOW for individual links.
  • NOARCHIVE instructs the crawler not to keep the page in the engine's public index. With Google, this prevents a cached copy of the page from being available in search results.

Preventing crawlers from following undesirable links can help combat automated spam. This method is endorsed by all major engines. Adding a rel="nofollow" attribute to a link allows control on an individual link basis. It can be used to reference external content while not endorsing it, and overcomes the necessity to exclude the crawler from all links on a page.
<a href="http://www.badsite.com" rel="nofollow">

You can use all the above access restriction methods in Magnolia:
  • Editing a robots.txt file is vastly simplified in that Magnolia produces clean, human-readable URLs such as http://www.magnolia-cms.com/support-and-services.
  • At a single-page level, templates set the robots meta element in the HTML header to "all" by default, allowing full crawler access. Options to change this behavior can be offered to editors with minimal configuration directly in AdminCentral. Customization would involve adding a field in the Page Info dialog definition and a minor tweak to the paragraph template.

Popular posts from this blog

Vanity URLs with Magnolia CMS

Image
Vanity URLs are short, easy to remember Web addresses used for campaigns, microsites and landing pages. They are a key Internet marketing tool. In this post I whip out my vanity cane, clip on my grill and show you how to get vain with Magnolia CMS. A vanity URL typically contains a word or phrase that describes a specific target such as a campaign slogan. When a user visits the vanity URL they are redirected to the actual target site. The URL can be constructed using a: trailing path: www.shell.com/ eco-marathon domain prefix: experience .virgin-atlantic.com domain suffix: www.amazon. de entirely unique domain name: mynavyfuture.com The URL is often reproduced in print ads, hoping that the short and snappy look makes it easier to remember. Magnolia CMS provides two approaches to creating vanity URLs: 1) Virtual URI mapping is used to create path-based vanity URLs. In this mechanism a trailing path is mapped to an actual page. In the configuration example below, www.example.com/winter
Read more

Search Engine Optimization

Image
This is the first post in a series on search engine optimization (SEO). I will be looking at key SEO factors identified by the SEOmoz.org consulting group and pitting them against Magnolia CMS , a Web content management system. This series should be interesting to online marketers, people employing web communications, and anyone looking to increase their site rank. Download all posts in a single PDF. What is SEO? The purpose of SEO is to enhance the volume and quality of traffic directed to a website via search engine results. Search engines aim to provide users with the most relevant results, while webmasters attempt to tweak their sites in order to move it up in the rankings and attract profitable, organic traffic. SEO began in the early 1990s, and quickly degenerated into a tug of war between search engines and webmasters. For each new weighting factor introduced by the engines, crafty webmasters invented means to circumvent it. As a result, many undeserving sites achi
Read more

Keywords in page titles and meta elements

Image
This post belongs to a series about  search engine optimization  (SEO) with Magnolia CMS. Today we look at inserting keywords in page titles and meta elements. Page titles are an extremely important SEO factor. Search engines give titles considerable algorithmic weight and use them as headings on search results. Titles should include keywords and, where appropriate, your company name for branding purposes: <title>Magnolia - Finance, Insurance & Banking</title> Description meta elements on the other hand do not add much to page relevance, but should not be neglected: <meta name="description" content="Magnolia's largest group of Enterprise Edition customers is from the..."> Their content often appears below the title in search results: The relevance of keywords stipulated in the keyword tag once had great impact, but this is no longer true. Actual page content is now far more relevant. Still, a keyword tag should be included:
Read more