One of the most regrettable yet frequent accidents during website redesigns and migrations happens like this: while developing on a staging server (such as staging.example.com), the development team selects “Discourage search engines from indexing this site” in the global settings to prevent search engines from indexing the staging website prematurely. On launch day, the operations team synchronizes the staging database and code to the production domain in a single step, but forgets to turn off noindex. Within a week of the new website's launch, dozens of core keywords that previously ranked on the first page disappear from both Baidu and Google.

Website

To prevent this kind of accident, the following five checks must be completed within 30 minutes of switching the domain's DNS records to production:

Config

1. Five steps to thoroughly check for leftover indexing restrictions

Site check

  1. Check the <meta name="robots"> tag in the page's HTML source Open the production website's homepage, service pages, case study pages, and article detail pages in a private browsing window. Right-click and select “View Page Source” (shortcut: Ctrl+U / Cmd+Option+U), then search for noindex or none. If you find <meta name="robots" content="noindex, nofollow">, immediately clear the relevant checkbox in the CMS admin panel (for example, “Settings -> Reading -> Search engine visibility” in WordPress, or the global SEO settings in a custom admin panel).
  2. Check X-Robots-Tag in the server's HTTP response headers Sometimes the HTML source contains no noindex, but Nginx / Apache or a CDN edge node returns X-Robots-Tag: noindex in the HTTP headers. Open the browser's developer tools (F12), switch to the Network panel, refresh the page, click the first document request, and check whether this field appears under Response Headers.
  3. Verify the contents of the /robots.txt file Go directly to https://www.your-domain/robots.txt. If it contains these two lines:
    User-agent: *
Disallow: /
    The entire website is blocked from crawling! In production, robots.txt should allow crawling of the root directory (Allow: /), block only admin paths (such as /admin/ and /api/), and provide the correct Sitemap: URL.
  4. Check whether canonical tags still point to the staging domain Search the source for rel="canonical". If a production page's canonical URL still contains https://test.xxx.com/ or a local development address such as http://localhost:3000/, search engines will likewise stop indexing the production domain.
  5. Verify using Google Search Console's “Test Live URL” Enter the production homepage URL in the search box at the top of GSC, click “Test Live URL” on the right, and confirm that the result displays a green “URL is available to Google” message for complete peace of mind.