{"id":26034,"date":"2024-10-24T15:55:56","date_gmt":"2024-10-24T05:55:56","guid":{"rendered":"https:\/\/digitalnomadshq.com.au\/?post_type=glossary&amp;p=26034"},"modified":"2024-10-24T15:55:56","modified_gmt":"2024-10-24T05:55:56","slug":"x-robots-tag","status":"publish","type":"glossary","link":"https:\/\/digitalnomadshq.com.au\/glossary\/x-robots-tag\/","title":{"rendered":"X-Robots-Tag"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">What is the X-Robots-Tag?\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When a browser or a search engine bot requests a resource from your server, the server sends back an HTTP response, which can include various headers. The X-Robots-Tag is one of those headers that gives specific instructions on how search engines crawl and index their pages.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While many of us are familiar with the traditional robots.txt file or the robots meta tag that you can place in the HTML of a webpage, the X-Robots-Tag works a bit differently. It allows you to apply indexing instructions at the HTTP header level, which is useful for non-HTML files like PDFs, images, and other media types.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An example of what the HTTP response with the X-Robots-Tag may look\u202flike:&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">HTTP\/1.1 200 OK&nbsp;<br>Date: Tue, 23 Oct 2024 10:32:20 GMT&nbsp;<br>Content-encoding: gzip&nbsp;<br>(\u2026)&nbsp;<br>X-Robots-Tag: noindex&nbsp;<br>(\u2026)&nbsp;<\/p>\n\n\n\n<div style=\"height:100px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Is the X-Robots-Tag Important?\u00a0<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The X-Robots-Tag provides a level of control that traditional methods can\u2019t always achieve. For example, let\u2019s say you have a downloadable PDF that you don\u2019t want to appear in search results. By using the X-Robots-Tag, you can set it to <strong>noindex, nofollow<\/strong> for that specific file. This way, search engines will crawl the page but recognise that they should neither index the document or follow any links within it.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here are a few reasons why it\u2019s important:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Enhanced Control<\/strong>: It gives you more precise control over how search engines interact with different types of content on your site.\u00a0<\/li>\n\n\n\n<li><strong>Preventing Duplicate Content<\/strong>: If you have multiple versions of the same content, you can use the X-Robots-Tag to prevent indexing of duplicates, which can help avoid penalties from search engines.\u00a0<\/li>\n\n\n\n<li><strong>Optimising Crawl Budget<\/strong>: By instructing search engines not to crawl certain pages or resources, you can ensure that your crawl budget is used effectively on your most important content.\u00a0<\/li>\n<\/ul>\n","protected":false},"featured_media":0,"parent":0,"template":"","meta":{"_acf_changed":false,"content-type":""},"class_list":["post-26034","glossary","type-glossary","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary\/26034","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary"}],"about":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/types\/glossary"}],"wp:attachment":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/media?parent=26034"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}