Start Monitoring Free
✦ Free SEO tool

X-Robots-Tag Checker

Read the X-Robots-Tag header of any URL, on every hop of a redirect, and see what each directive means for indexing.

Fetching and analysing the page…

5 of 5 free checks left today. Create a free account for unlimited checks.

X-Robots-Tag Checker - results and guide

What the X-Robots-Tag Checker checks

  • Every X-Robots-Tag header in the response
  • Directives per crawler, such as googlebot: noindex
  • noindex, nofollow, noarchive, nosnippet, noimageindex and the max-* limits
  • Unrecognised directives that crawlers will ignore
  • The header on each redirect hop, not only the final page
  • Conflicts with the robots meta tag in the HTML
  • The full response headers

What is the X-Robots-Tag header?

X-Robots-Tag is an HTTP response header that carries the same instructions as the robots meta tag: X-Robots-Tag: noindex, nofollow. Because it travels with the response instead of sitting in the HTML, it works for any file type. It is the only way to keep a PDF, an image, a video or a feed out of search results.

A directive can be limited to one crawler by prefixing its name, for example X-Robots-Tag: googlebot: noindex. Several headers may be sent at once and all of them apply.

Why check it?

The header is invisible in the browser and in the page source. It is added in the web server, the CDN or the application, often by a rule covering a whole folder or file type. A noindex meant for a staging host or for PDFs can end up on every page of a live site, and nothing looks wrong until traffic drops.

The header only works if the crawler can fetch the URL. If robots.txt blocks it, the header is never seen. For a complete answer on whether a page can be indexed, including meta tags and robots.txt, use the Noindex Tag Checker.

How to use the X-Robots-Tag Checker

  1. Enter the address of the page you want to check. You can leave out https://.
  2. Press Check header. The page is fetched from our servers, the way a search engine crawler would fetch it.
  3. Read the findings from top to bottom: problems first, then warnings. Each one says what to change.

Common problems and how to fix them

noindex appears on pages that should rank

Search the server configuration for X-Robots-Tag: Header set lines in Apache, add_header in Nginx, and response header rules in the CDN. Remove or narrow the rule.

PDFs show up in Google

Add the header for that file type. In Apache: a FilesMatch block for \.pdf$ with Header set X-Robots-Tag "noindex".

The header and the meta tag say different things

The stricter one wins, so the result may not be what you intended. Set the rule in one place.

Frequently asked questions

What is the difference between X-Robots-Tag and the robots meta tag?
They accept the same directives. The meta tag lives in the HTML of a page. The header is part of the HTTP response and so works for non-HTML files too.
How do I see the X-Robots-Tag header?
Enter the URL above, or look at the response headers in your browser's developer tools under the Network tab.
Does Google support X-Robots-Tag?
Yes. Google and Bing both honour it, with the same directives as the robots meta tag.
Can I set different rules for different bots?
Yes. Prefix the directives with the crawler's name, such as googlebot: noindex, and send a separate header for each crawler.

See all 27 free SEO tools →

Monitor Today. Stay Ahead.

SEO gets visitors to your site. Uptime keeps them there.

Monitor 5 websites free forever and get alerted the moment one goes down. No credit card required.

Start Monitoring Free Run a free AI check