Robot Using Laptop

Why Doesn’t Googlebot Crawl Enough Pages on Every Website?

If you’re wondering why search giant Google doesn’t always seem to crawl enough web pages, you’re not alone. Indeed, the industry juggernaut’s Search Advocate, John Mueller, was asked this very question in a Google Office Hours SEO hangout.

Budgetary considerations

Google’s crawler, Googlebot, goes from one web page to another, indexing them for ranking purposes. However, the internet is obviously huge and it’s the search engine’s policy only to index higher-quality web pages. It clearly wouldn’t be physically possible for Google to index every single URL.

Google says: “The amount of time and resources we devote to crawling a site is commonly called the site’s crawl budget. This is the set of URLs that GoogleBot can and wants to crawl.”

That doesn’t necessarily mean that everything crawled is indexed and if your site is of a lower quality, Googlebot may not crawl it.

As Mueller puts it: “We’ll be … more cautious about crawling and indexing (pages) until we’re sure the quality is good.”

Other relevant factors

Here are a couple of the other factors affecting the number of pages Google crawls:

  • A shared server may not be able to serve pages quickly enough to Google, since other sites using the server may be slowing its progress.
  • Rogue bots affecting the server could also be slowing your website down.

It’s actually also worth checking out of hours, because Google may be crawling outside the regular working day. Of course, these issues will affect you in particular if you have a very large website, which you update regularly. (As, of course, you should be doing.) However, if these don’t apply to you, frequent checking of your index coverage and updating your sitemaps should be enough to keep Google happy.

Best practices

The good news is that there are things you can do to optimise crawling efficiency:

  • Manage your URL inventory by consolidating duplicate content, blocking crawling of any links you don’t wish to be indexed and keeping sitemaps updated.
  • Monitor crawling of your website and monitor for availability issues to make it as efficient as possible.
  • Make sure your pages are efficient to load.
  • Ensure any updates are crawled quickly enough.

Talk to us

At Front Page Advantage, we’ve operated highly successfully in our industry for years. If you have any concerns about Google’s crawling of your website, talk to us before you talk to anyone else. Whatever the size of your business and whatever your industry, we’d be delighted to have an initial, no-obligation chat to discuss how to maximise Google crawling of your webs pages. This will allow you to make an informed decision regarding next steps.

Get in touch with a member of the friendly, professional team today.