Staging is hidden and production is not

This is the classic way to lose a site's traffic, and it happens in both directions.

The staging copy gets indexed. Every page of it is a duplicate of the real site, and Google has to choose between them. Sometimes it chooses the one at staging.example.com that nobody can buy anything from.

The `noindex` follows the site to production. A staging site is usually kept out of search with a noindex tag or a Disallow: / in robots.txt. On launch day the files are copied across, and the switch that was protecting the staging site now hides the real one. The site goes live, everything looks perfect, and it simply never appears in search — which nobody notices for weeks, because who searches for a site they can already see.

We check production continuously: indexing.noindex and indexing.robots-blocked are exactly this, watched every scan. What we cannot see is the staging copy, because we don't know where it is.

Protect staging with HTTP basic authentication, not with noindex. A password cannot be copied to production by accident and cannot be ignored by a crawler that doesn't feel like obeying it.

If you must use noindex, make it come from an environment variable rather than a file that is deployed:

X-Robots-Tag: noindex

sent by the staging server alone. Then, on launch day, check production with View source — look for noindex and read /robots.txt — before you tell anyone it is live.

Tick this when staging needs a password and you have read production's own robots.txt with your own eyes.