A meta robots tag is an HTML tag that gives search engine crawlers<\/a> instructions on how to index, crawl, and display a page in search results. It\u2019s placed within the Here\u2019s an example in code:<\/p>\n\n\n\n The meta robots tag is mainly used to control page indexing and crawling and can be found in the page’s HTML code. For pages without an HTML head section or for resources like images or PDFs, the X-Robots tag is used instead.<\/p>\n\n\n\n Unlike the meta robots tag, the X-Robots tag is an HTTP header sent with the webpage\u2019s response.<\/em><\/p>\n\n\n\n You can use the Ahrefs SEO Toolbar browser extension to check a page’s meta robots tag – The “indexability” section shows crawler directives based on both meta tags and HTTP headers.<\/p>\n\n\n\n It’s important to remember that search engine crawlers can only follow the instructions in the meta robots tag if the page is allowed to be crawled according to the rules in the Beyond controlling page indexing and crawling, the meta robots tag can also instruct search engines not to show a text snippet or video preview in search results or to avoid caching the page.<\/p>\n\n\n\n Webmasters often want to prevent certain pages from appearing in search engines, such as those with thin content, PPC landing pages, or admin and “thank you” pages. The meta robots tag is the most reliable way to achieve this by using the \u201cnoindex\u201d directive.<\/p>\n\n\n\n Additionally, the “nofollow” attribute in the meta robots tag ensures that all links on the page are treated as \u201cnofollow\u201d by search engine crawlers like Googlebot,<\/a> meaning no link equity<\/a> or “link juice<\/a>” is passed to the linked pages.<\/p>\n\n\n\n This tag can also control how a page is indexed and displayed to users – For example, a “nosnippet” directive can be added to stop search engines from showing a text snippet from the page in search results.<\/p>\n\n\n\n Setting up the meta robots tag in WordPress is straightforward with SEO plugins like Yoast and RankMath. Here\u2019s how to do it:<\/p>\n\n\n\n Using Yoast SEO Plugin<\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n Using the RankMath Plugin<head><\/code> section of the webpage.<\/p>\n\n\n\n<meta name=\"robots\" content=\"noindex, nofollow\"><\/em><\/code><\/p>\n\n\n\nrobots.txt<\/code> file. If you want a page to be excluded from search results, the page must be crawlable<\/a> so that search engines can read the “noindex” directive.<\/p>\n\n\n\nImportance of Meta Robots Tag<\/h3>\n\n\n\n
Setting Up the Meta Robots Tag<\/h2>\n\n\n\n
\n
<\/strong>To noindex pages with RankMath, go to the Advanced tab and select the “No Index” option.<\/p>\n","protected":false},"featured_media":0,"parent":0,"template":"","meta":{"_acf_changed":true,"content-type":""},"class_list":["post-25537","glossary","type-glossary","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary\/25537","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/glossary"}],"about":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/types\/glossary"}],"wp:attachment":[{"href":"https:\/\/digitalnomadshq.com.au\/wp-json\/wp\/v2\/media?parent=25537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}