Search results for robots

If you ve ever wondered how to get a little better control over what parts of your web site get crawled by the search engines how they crawl your pa read more »
Discuss   Add this link to...  Tell a friend   Bury
We've improved Webmaster Central's robots.txt analysis tool to recognize sitemap declarations and relative urls. Earlier versions (Read more...) read more »
Discuss   Add this link to...  Tell a friend   Bury
The Google Robots.txt Generator tool has been unveiled inside of Webmaster Tools to help users allow or disallow crawling of files and directories. read more »
2 Comments   Add this link to...  Tell a friend   Bury