How to create a robots.txt file with sitemap location

June 9, 2009 09:34 by gareth

 

Theres nothing new about a robots.txt file, but I have started using robots.txt validators that will not validate unless an XML sitemap is specified.

This is very good practice as the XML sitemap will get picked up by all crawlers that support the tag 'Sitemap' within your robots.txt file (Google and MSNBot do for starters).

So using http://www.sitemaps.org/protocol.php#submit_robots as a reference my example robots.txt file will look like this:

Sitemap: http://www.example.com/sitemap.xml

User-agent: *
Disallow: /directory-x
Disallow: /directory-y

The blank link between the Sitemap tag and the User-agent tag is essential as this can cause problems for some crawlers.

Happy SEO!

 


Currently rated 5.0 by 1 people

  • Currently 5/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Related posts

Comments

Add comment


 

[b][/b] - [i][/i] - [u][/u]- [quote][/quote]



Live preview

September 3. 2010 14:04

Calendar

September 2010
SuMoTuWeThFrSa
2930311234
567891011
12131415161718
19202122232425
262728293012
3456789

Tags