What is AI crawlers?
Also called: GPTBot, ClaudeBot, PerplexityBot
The short answer
AI crawlers are bots run by AI companies that fetch web pages to train models or to answer questions live; robots.txt can allow or block each one.
AI crawlers, explained
Blocking them keeps content out of some AI systems but also out of their answers. Most businesses that want to be recommended allow the answer-time crawlers.
Questions people ask about AI crawlers
What is AI crawlers?
AI crawlers are bots run by AI companies that fetch web pages to train models or to answer questions live; robots.txt can allow or block each one.
Related terms
robots.txt is a file at a site's root that tells crawlers which paths they may or may not crawl, and where the sitemap is.
llms.txtllms.txt is a proposed plain-text file at a site's root that gives AI models a concise, curated guide to the site's most important content and links.
AI citationAn AI citation is a link or reference an AI answer gives to the page it drew information from, the visible credit that sends traffic and trust to the source.
AEOAEO, answer engine optimisation, is structuring a website so AI assistants and search features quote it as the answer, not just list it as a link.
Part of the Web Encyclopedia by Apex Flow Digital. Definitions are general knowledge, kept current, with no invented statistics.