What is Crawling?
Also called: Crawler, Spider, Bot
The short answer
Crawling is how search engines discover pages: automated bots follow links and sitemaps, fetch pages and pass them on to be analysed and indexed.
Crawling, explained
Pages no link points to are hard to find. Internal links and sitemaps make sure crawlers reach everything that matters.
Questions people ask about Crawling
What is Crawling?
Crawling is how search engines discover pages: automated bots follow links and sitemaps, fetch pages and pass them on to be analysed and indexed.
Related terms
Indexing is when a search engine stores a crawled page in its database so it can appear in results; a page that is not indexed cannot rank.
robots.txtrobots.txt is a file at a site's root that tells crawlers which paths they may or may not crawl, and where the sitemap is.
Internal linkingInternal linking is linking between pages on your own site, which helps visitors move on and tells search engines how pages relate and which matter most.
SEOSEO is the practice of making a website easier for search engines to crawl, understand and rank, so it appears for the searches its audience actually makes.
Part of the Web Encyclopedia by Apex Flow Digital. Definitions are general knowledge, kept current, with no invented statistics.