Error Alerts at once indexing.robots-blocked

robots.txt stops search engines reading the page

Why it matters

A Disallow rule in robots.txt stops crawlers fetching the page, so they never see its title, text or tags. The address can still show up in results, as a bare link with "No information is available for this page", which is worse than not appearing. It also hides any noindex on the page, because a crawler that can't fetch the page can't read it.

Source: Introduction to robots.txt. We test the rules for Googlebot, falling back to *, the way Google documents matching.

How to fix it

Open /robots.txt and remove or narrow the rule we matched. A leftover from staging is the usual cause:

User-agent: *
Disallow: /

To let everything be crawled, leave Disallow: empty. To keep a page out of search results, don't use robots.txt: allow it to be crawled and give it a noindex meta tag instead.

Examples

These are the rule's own test pages, and what it says about each. They are run on every change to the rule book.

They differ only in what the server answered:

blocks this page (Disallow: /)

Errorrobots.txt blocks search engines from this page (Disallow: /), so they can't read its content.

fail-disallow-all
allows this page

Passedrobots.txt lets search engines read this page.

pass-allowed