Theres nothing new about a robots.txt file, but I have started using robots.txt validators that will not validate unless an XML sitemap is specified.
This is very good practice as the XML sitemap will get picked up by all crawlers that support the tag 'Sitemap' within your robots.txt file (Google and MSNBot do for starters).
So using http://www.sitemaps.org/protocol.php#submit_robots as a reference my example robots.txt file will look like this:
Sitemap: http://www.example.com/sitemap.xml
User-agent: *
Disallow: /directory-x
Disallow: /directory-y
The blank link between the Sitemap tag and the User-agent tag is essential as this can cause problems for some crawlers.
Happy SEO!
Currently rated 5.0 by 1 people
- Currently 5/5 Stars.
- 1
- 2
- 3
- 4
- 5