What is robots.txt?
The short answer
robots.txt is a file at a site's root that tells crawlers which paths they may or may not crawl, and where the sitemap is.
robots.txt, explained
It controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.
Questions people ask about robots.txt
What is robots.txt?
robots.txt is a file at a site's root that tells crawlers which paths they may or may not crawl, and where the sitemap is.
Related terms
Crawling is how search engines discover pages: automated bots follow links and sitemaps, fetch pages and pass them on to be analysed and indexed.
XML sitemapAn XML sitemap is a machine-readable file listing a site's URLs, and when each last changed, submitted to search engines to help them discover pages.
NoindexNoindex is an instruction, set in a meta tag or HTTP header, that tells search engines not to show a page in their results.
AI crawlersAI crawlers are bots run by AI companies that fetch web pages to train models or to answer questions live; robots.txt can allow or block each one.
Part of the Web Encyclopedia by Apex Flow Digital. Definitions are general knowledge, kept current, with no invented statistics.