Skip to main content
Legia

SEO & Digital Marketing

What is Crawling?

Also called web crawling, spidering, crawl.

Short definition

Crawling is the process by which search engine robots, such as Googlebot, discover and download web pages by following links and sitemaps; pages that have not been crawled cannot yet be indexed or shown in search results.

How robots find pages

There is no central list of every page on the web. Googlebot finds new URLs from links on pages it already knows, from sitemaps submitted by site owners, and from other sources. After downloading a page, Google also renders it, including running JavaScript, so the content looks the way it does in a regular browser.

What gets in the way of crawling

A slow server or frequent errors make Google reduce its crawl rate to avoid overloading the site. Overly broad robots.txt rules can block important sections. Pages that can only be reached through a search box, or links built with scripts instead of regular HTML link elements with an address, are hard to discover. Content that only appears after login will not be crawled.

Crawl budget and company sites

Crawl budget, meaning how many pages Google crawls within a given period, is mainly a concern for very large sites or sites whose content changes very quickly. For company websites with tens to thousands of pages, the problem is more often pages that are not linked from anywhere, or server errors. The crawl stats report in Search Console helps you see this. It shows the number of requests per day, server response time, and the types of errors Googlebot ran into.

Related Legia service SEO Technical fixes, content structure, and local search, measured through to incoming inquiries.