X-Robots-Tag Checker
Inspect the HTTP headers of any URL (HTML pages, PDFs, images) to check for X-Robots-Tag indexing directives.
Try URLs for HTML pages, PDFs, images, or API endpoints.
Fetching HTTP headers... please wait.
Header evaluation complete.
What is the X-Robots-Tag?
The `X-Robots-Tag` is an HTTP header sent by your web server that functions similarly to the HTML `<meta name="robots">` tag. However, while meta tags only work on HTML documents, the `X-Robots-Tag` can be applied to any file type.
This makes it the only effective way to prevent search engines from indexing non-HTML files such as PDFs, images (JPG, PNG), videos, or plain text documents.
Protecting Non-HTML Files
If you have an internal PDF document or a private image, putting a rule in `robots.txt` might stop crawling, but Google can still index the URL if someone links to it. Only an `X-Robots-Tag: noindex` will force Google to remove the PDF from search results.
Server-Level Control
By configuring your Apache or Nginx server, you can apply an `X-Robots-Tag` conditionally across an entire directory (e.g., forcing `noindex` on all `.doc` files or a specific staging environment) without modifying the application code.
Frequently Asked Questions
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
Check Out Other Popular Tools
FICO Score Estimator
Estimate your FICO credit score for free using official scoring factors. Understand how payment history, utilization, and credit length affect your credit health.
Convert Letters to Binary Online - Smart Text Converter
Instantly convert letters to binary code and vice versa. Smart auto-detection, UTF-8 support, and advanced formatting options. The best free online binary translator.
Palindrome Text Generator
Generate a palindrome from any text using character mirror, word mirror, or check if your text is already a natural palindrome. 100% private, runs in your browser.
Was this tool helpful?
Comments
Loading comments...