A robots.txt file works a lot like a Sitemap in that it helps search engines understand how to access and review your site, but unlike the Sitemap, which lists pages to be accessed, the robots.txt file tells the search engine which files or directories to exclude from its review. A robots.txt file may also be used to show search engines where the Sitemap can be found.
