Using the robots.txt file to tell search engines about your sitemap

There was a good piece of news announced in April by Google and the other main search engines – Yahoo! and MSN – concerning further enhancements of the recent Sitemap protocol. This standardised method of supplying these search engines with a recognised format to help with the indexing of a website has now been extended to the robots.txt file as well.

We reported in December about the significant step forward made by the ‘big 3′ search engines in agreeing a standardised protocol for submitting a Sitemap document – in a number of different formats – which would enable these search tools to index pages from a website more efficiently and more often, if required.

Now Google and the others (who have also been joined by Ask) have taken this another step forward by announcing a simple addition that can be made to a robots.txt file, which will enable these Sitemap files to be identified and indexed by the indexing ‘spiders’. Although not essential, the robots.txt file is something that the automated ‘spiders’ sent out by the main search engines will look for when they first arrive on a website’s server, as it should indicate which sections of the site can, or can’t, be indexed. This can help to streamline the indexing process and is something we recommend that every website should include ‘ you can find out more about this file here.

Now, the search engines that use the standardised Sitemap protocol have extended the service to allow website managers to insert a command in the robots.txt file that will direct the spiders to the Sitemap file. This means that you don’t necessarily need to register to use Google’s Webmaster tools, for example, (although this can still be the best option) but you can just edit your robots file to include the location of the Sitemap file and then let the search engines do the rest.

There’s more information about this on the official Sitemaps protocol website.

Leave a Reply