{"id":24700,"date":"2024-08-23T16:50:11","date_gmt":"2024-08-23T06:50:11","guid":{"rendered":"https:\/\/digitalnomadshq.com.au\/?post_type=glossary&#038;p=24700"},"modified":"2026-03-30T23:29:19","modified_gmt":"2026-03-30T13:29:19","slug":"crawl-budget","status":"publish","type":"glossary","link":"https:\/\/digitalnomadshq.com.au\/glossary\/crawl-budget\/","title":{"rendered":"Crawl Budget"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">What is Crawl Budget?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">There are over a billion websites across the globe today. And this number continues to grow. In fact, data suggests that over 250\u202f000 new websites are created everyday. For Google, that\u2019s a lot of ground to cover!&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To manage this massive task, Google uses a &#8220;crawl budget&#8221; to determine how many pages on each website it can crawl and index within a given time frame. Put simply? It\u2019s the amount of attention and resources Google\u2019s crawlers can allocate to your site.<\/p>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">How Does It Work?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A crawler, like Google Bot, will systematically go through a list of URLs on a site. It will start with URLs it knows about, following links withing the site and finding new pages as it does this. As they find new links on these pages, they add it to the list of pages to crawl next.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, with so many pages on the internet, crawlers don\u2019t have unlimited time or resources, so it prioritises certain pages based on factors like content quality, freshness, and relevance.<\/p>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">What Determines Crawl Budget?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A website\u2019s \u201ccrawl budget\u201d is determined by a number of factors, including the size of your site, your site\u2019s health (think loading speed and server response times) and how frequently your content is updated. Some of the most common factors that determine budget include:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Importance of webpage: <\/strong>As web crawlers can\u2019t crawl the entire internet, it decides which pages to crawl based on how many pages are linked to that page, how many visitors it gets and other factors that might indicate that it is relevant or important to users.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Updated\/New Webpages: <\/strong>If a site regularly adds new pages or updates existing ones, crawlers are likely revisit these pages to ensure they are up to date.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Robots.txt Requirements: <\/strong>A robots.txt file guides crawlers on which pages they should or shouldn\u2019t crawl. If properly configured, it can help focus crawl budget on the pages that matter most.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Size of Site: <\/strong>Larger sites naturally require more crawl budget simply because they have more pages to be indexed. However, if a large portion of these pages are low-quality or duplicate content, it can waste the budget and leave important pages uncrawled.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Health of Site: <\/strong>The speed and reliability of a site also play a role in budget allocation. If a site is slow or frequently encounters errors, crawlers might spend less time indexing these pages, reducing the crawl budget.&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Crawl limit:<\/strong> This is how much a website can handle and is impacted by server capabilities craw limit set by site owner in <a href=\"https:\/\/digitalnomadshq.com.au\/glossary\/google-search-console\/\">Search Console<\/a> etc.\u00a0\u00a0<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">While there is no exact details on your website\u2019s crawl budget, you can monitor craw activity in Search Console\u2019s <a href=\"https:\/\/search.google.com\/search-console\/settings\/crawl-stats\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Crawl Stats Report<\/a>.&nbsp;&nbsp;<\/p>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Why Does Crawl Budget Matter?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When you have a website, you want to ensure that your most important pages (new, old and refreshed) are discovered, crawled, and indexed by search engines; <em>as quickly as possible.&nbsp;<\/em>&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, not all pages on your website are created equal. Some pages are more important than others, and if search engines run out of crawl budget before getting to these pages, they might not be indexed or updated in search results as quickly as you\u2019d like.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While not necessarily a big concern for small, or new websites, efficient use of your crawl budget can help ensure that search engines focus on your most valuable content. This can improve your site\u2019s visibility in search results ensuring that users find the pages that matter most to them when searching for relevant information.&nbsp;<\/p>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">When Is Crawl Budget an Issue&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">As <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">outlined by Google<\/a>, craw budget allocation may become an issue if you have:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Large sites (1 million+ unique pages) with content that changes around once a week.&nbsp;&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Medium or larger sites (10,000+ unique pages) with content that changes daily.&nbsp;&nbsp;<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sites with URLs classified by Search Console as\u202f\u201cDiscovered &#8211; currently not indexed\u201d&nbsp;<\/li>\n<\/ul>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">How To Optimise Your Crawl Budget&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">There are a number of ways that you can increase your website\u2019s crawl budget. These include:&nbsp;<\/p>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li><strong>Remove or Consolidate Low-Value Pages:<\/strong> If you have pages that don\u2019t add much value, consider removing them or merging them with more relevant content.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li><strong>Improve Page Load Times:<\/strong> Faster pages mean crawlers can get through more of your site in less time. Compress images, minify code, and use caching to boost speed. (<em>this isn\u2019t just great for crawl budget, but also fantastic for users!)<\/em>&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li><strong>Fix Broken Links: <\/strong><a href=\"https:\/\/digitalnomadshq.com.au\/glossary\/broken-link\/\" target=\"_blank\" rel=\"noreferrer noopener\">Broken links<\/a> lead crawlers to dead ends, wasting valuable crawl budget. Regularly check for and fix any broken links on your site to ensure there is a clear link between pages across your site.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol start=\"4\" class=\"wp-block-list\">\n<li><strong>Optimise Internal Links:<\/strong> Googlebot prioritises pages with more backlinks and internal links and this tells the crawler that the page is important. By optimising your internal and external link building strategy you can guide crawlers to key pages.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol start=\"5\" class=\"wp-block-list\">\n<li><strong>Use a Sitemap:<\/strong> Submitting a sitemap to search engines helps them find your important pages more easily, ensuring they get crawled and indexed.&nbsp;<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"featured_media":0,"parent":0,"template":"","meta":{"_acf_changed":true,"content-type":""},"class_list":["post-24700","glossary","type-glossary","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary\/24700","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary"}],"about":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/types\/glossary"}],"wp:attachment":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/media?parent=24700"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}