Article

Troubleshoot Google Search crawling errors

Google Search Central · Google Search Central (Documentation) Free Intermediate

When you want to know why Google is not showing your pages, the most honest answer comes straight from Google, not from a blog guessing at it. This is the official crawling and indexing troubleshooting doc, covering the self-inflicted errors (soft 404s, blocked resources, server availability, wasted crawl budget) that quietly keep founder sites out of search. It is a reference to return to, not a one-time read.

• If Googlebot cannot reach a page (blocked resource, server returning errors, wrong HTTP status), nothing else about your SEO matters, the page simply will not rank.
• Soft 404s (a page that looks broken but returns a success code) are a common silent killer, Google quietly drops these from the index.
• Do not use robots.txt to hide pages you want kept out of search, it blocks crawling but not indexing, use noindex or a login instead.