{"id":25877,"date":"2024-10-04T16:22:53","date_gmt":"2024-10-04T06:22:53","guid":{"rendered":"https:\/\/digitalnomadshq.com.au\/?post_type=glossary&#038;p=25877"},"modified":"2025-12-22T16:56:00","modified_gmt":"2025-12-22T06:56:00","slug":"robots-txt","status":"publish","type":"glossary","link":"https:\/\/digitalnomadshq.com.au\/glossary\/robots-txt\/","title":{"rendered":"Robots.txt"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">What is Robots.txt?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Sometimes, you don\u2019t want web crawlers to access specific URLs on your website. To do so, you can use a <strong>robots.txt<\/strong> file to block certain pages or sections of your website. This file gives instructions to search engine bots about which parts of your site they should or shouldn&#8217;t crawl.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How Does Robots.txt work?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The robots.txt file is a simple text file located in the root directory of your website. When a crawler visits your website, it checks the robots.txt file for any guidelines before it starts crawling.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Note: Not all crawlers are \u201cgood\u201d and may ignore these instructions. These are usually for \u201cscraping\u201d where they extract different data from your site without permission.<\/em>&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can tell crawlers which pages they are allowed to visit and which ones to avoid by specifying directives like \u201callow\u201d or \u201cdisallow.\u201d For example, if you have pages you don\u2019t want appearing in search results\u2014such as admin sections or duplicate content\u2014you can block them by adding the appropriate instructions in your robots.txt file.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, it&#8217;s important to note that while most crawlers will adhere to the rules specified by robots.txt, the file should only be used to reduce and optimise crawler resources and <em>not<\/em> control the indexing of pages. This is because a disallowed URL can still be indexed if it is found by an external link.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Robots.txt Features:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>user-agent:<\/strong> The crawler the rules will apply\u202fto\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>disallow:<\/strong> A path that must not be crawled or accessed\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>allow:<\/strong> An optional field that says what path that can be crawled\u00a0\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>sitemap:<\/strong> An optional field that describes the location of the sitemap file\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>crawl-delay:<\/strong> An optional field that controls the crawling speed. This is, however, not supported by GoogleBot.\u00a0\u00a0<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">For Example:&nbsp;&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>User-agent: Googlebot\u00a0\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Disallow: \/admin\/\u00a0\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Allow: \/admin\/allowed-page\/\u00a0\u00a0<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sitemap: <a href=\"https:\/\/www.example.com\/sitemap.xml\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">https:\/\/www.example.com\/sitemap.xml<\/a>\u00a0<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">So, this example blocks Googlebot from accessing the \/admin\/ directory but allows it to crawl the \/admin\/allowed-page\/.&nbsp;&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why Are Robots.txt Important?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The primary benefit of a robots.txt file is that it allows you to optimize the crawl budget of your website. Search engines have a limited amount of time to crawl each site, so by using robots.txt, you can direct them away from pages that aren\u2019t important or for public access. For CMS like WordPress, the admin pages are automatically blocked by crawlers.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Important note: It is not recommended to rely solely on the robots.txt file to control the indexing of pages. As we\u2019ve explained, robots.txt is more of a guideline that a rule.&nbsp;&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Does My Website Need One?&nbsp;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Not every website absolutely requires a robots.txt file. If you have a smaller website with just a few pages, or if you don\u2019t have any specific content that you need to block from search crawlers, it may not be necessary.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In many cases, search engines can effectively crawl your site without it. However, if you have large volumes of content, duplicate pages, or sensitive areas you\u2019d like to keep private (like admin panels or staging environments), implementing a robots.txt file can direct them to focus on more important pages and content.&nbsp;&nbsp;&nbsp;<\/p>\n","protected":false},"featured_media":0,"parent":0,"template":"","meta":{"_acf_changed":true,"content-type":""},"class_list":["post-25877","glossary","type-glossary","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary\/25877","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary"}],"about":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/types\/glossary"}],"wp:attachment":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/media?parent=25877"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}