Opens in a new tab
A woman holding a sheet of paper and a man using a laptop at a flooring showroom counter, with wooden samples in the foreground and natural window light.
SEO

Search Console blocked by robots.txt: what to check

An important page is live, but Google cannot crawl it. Confirm the robots.txt block, find the rule behind it and verify a targeted repair without opening every excluded URL.

Your new service page opens normally. Customers can reach it from the menu. Yet Search Console says it is Blocked by robots.txt, and it is not bringing in the search visits you expected.

Start by checking whether that block is intentional. A shop can have good reasons to restrict crawling of some URLs. The priority is to establish whether an important sales page has been caught by a rule meant for something else.

Google explains that robots.txt controls crawler access. It does not control whether a person can open a page, and a blocked URL can sometimes still appear in search results. Being able to browse the page yourself therefore does not settle the issue.

Choose the pages that actually need attention

Do not treat the total number of blocked URLs as a target to get down to zero. Start with pages that should help someone choose or buy: your core services, useful product categories, current products and substantial project information.

For each example, record the exact address, why it should appear in search and when the issue was noticed. Pick a few affected pages from different sections, plus an important page that appears unaffected. That gives your developer a useful comparison.

Imagine a flooring retailer has launched a fitting service. Its main shop pages are accessible to Google, but the new fitting pages sit inside a restricted directory. That calls for a focused check of that directory and its intended use, rather than a rewrite of every page on the website.

Keep deliberately excluded filter combinations and other unwanted URLs on a separate list. Their presence in the report may be expected.

Confirm the block on the exact URL

Inspect an affected address with Search Console’s URL Inspection tool. Compare Google’s last-crawl information with a live test. Check the date and Crawl allowed? field before treating the report as current.

Keep these questions separate:

  • Is crawling allowed? A robots.txt rule may prevent access.
  • Can the page be fetched? Access can fail for other reasons too.
  • Is indexing allowed? A page can contain a separate instruction excluding it from search.

One trap: a blocked page can show indexing as allowed because Google cannot read its noindex instruction. Treat that field in context.

If the live test finds no crawl block, record its time and investigate the remaining issue. Do not edit robots.txt to fix an old status. Passing a live test does not guarantee indexing.

Find the rule and the system supplying it

Ask your developer to retrieve the robots.txt file for the exact public website address being tested. Under Google’s robots.txt specification, rules apply to their particular host, protocol and port. Checking a non-www address does not establish what a www address serves.

The review needs to identify the crawler group and the matching path. For example, Disallow: / blocks all paths for the applicable group, unless a more specific allowance applies. A rule covering /services/ can catch a whole service section. Path matching is case-sensitive, and Google’s choice depends on rule specificity, not simply which line appears first.

Do not paste a generic replacement file over the existing one. Ask which rule actually blocks each example URL, and what else would change if it were removed or narrowed.

Trace the published file back to its source

Have the person maintaining the website establish whether the response comes from a physical file, the website platform, an extension or the hosting configuration. Request the setting or file responsible for the live response, rather than a screenshot of an editor that might not control it.

Then compare the timing with recent work. Was there a launch, migration, directory change or attempt to reduce crawler traffic? These are investigation leads, not proof of the cause.

If a restricted section has gained a legitimate sales purpose, make that decision explicit. The developer needs to know which pages should become accessible and which exclusions should remain.

Check what Google is meant to see

Before approving a change, decide the intended outcome for each affected page.

A public page that should attract customers

Remove or narrow the unintended crawl restriction, then check the page itself for any separate indexing exclusion. Also inspect its important supporting resources. Google’s robots.txt guidance warns against blocking resources whose absence makes a page harder for its crawler to understand.

For a service page, the acceptance check should include the actual offer, service information and route to enquire. Testing only the homepage will not demonstrate that the affected section works.

Do not assume a crawl block will reliably remove it. Google’s noindex documentation explains that Google must be able to access a page to see its noindex instruction. It also states that placing a noindex rule inside robots.txt is unsupported.

Ask your developer to choose the appropriate indexing control for the page’s purpose, and explain how Google will encounter it. Avoid adding conflicting restrictions and hoping one will take priority.

Content that should be private

Use proper access controls. Robots.txt is not a security boundary, as Google’s introductory guidance makes clear. If a supposedly private area is publicly accessible, deal with that access issue before considering its search appearance.

Make a targeted change and check the published result

A useful change request contains the affected addresses, their intended search visibility, the responsible rule and a small set of comparison URLs. Ask the developer to retain the previous configuration and describe the proposed change before applying it.

For the flooring example, the brief might be: allow the public fitting-service pages to be crawled while retaining the agreed exclusions elsewhere. Request evidence for the specific paths involved, rather than asking to “unblock Google everywhere”.

After the change, check the publicly served robots.txt response. A saved setting alone is not sufficient evidence. Test a page that should now be allowed, an unaffected sales page and a URL whose deliberate crawl restriction should remain.

Google’s instructions for updating robots.txt distinguish publishing the changed file from refreshing Google’s cached copy. If an old version is still being served publicly, resolve that delivery problem first.

Separate the robots.txt refresh from page recovery

Search Console’s robots.txt report shows fetched files, crawl information and processing issues. You can inspect the last fetched version and use the recrawl request when correcting an error or making a critical change, including unblocking important URLs.

That request concerns the robots.txt file. Google explicitly says it does not guarantee an immediate recrawl of the newly unblocked pages.

Once the corrected rules are available, run another live inspection of the affected page. Request indexing where appropriate, then record what has actually been established:

  1. The public robots.txt response contains the intended rules.
  2. The important page passes the live access check.
  3. Google later revisits the page under the corrected rules.
  4. The page’s indexing status and subsequent search performance can then be reviewed.

Give each stage its own date and evidence. This prevents a successful file edit being reported as recovered rankings or enquiries before either has happened.

What to ask your SEO or website supplier for

You should be able to get a plain answer to five questions: which pages were affected, which rule caused it, why the rule existed, what changed and how the result was checked.

Ask for a short follow-up plan as well. Who will check the later crawl and indexing status? Which important pages will be rechecked after the next website release? Keep the examples small enough that someone will actually review them.

If this surfaced during a rebuild, our guide to rebuilding a website without losing search traffic covers the wider handover. If Google has already crawled a page but has not indexed it, use our separate guide to Crawled – currently not indexed to investigate that different situation.

BuzzBoost’s SEO work includes technical crawlability and indexing checks. If a valuable page is blocked and you need a clear diagnosis, send us the affected address and what Search Console reports. We can help establish the cause and the next practical step.

Featured image: AI-generated editorial artwork, not a photograph of a real BuzzBoost office, client or result.

Bolt AI — BuzzBoost Digital author avatar
Written by Bolt AI