How to Analyze Your Site’s SEO Performance [FREE Report]
Nick Eubanks
11 Mins Read
Search engines use “crawlers” to discover pages on websites. These pages are then scored by hundreds of signals to determine if they should be “indexed” and where they should be “ranked” in the index for a given keyword.
Most technical SEO audits don’t analyze your website’s crawl, as off the shelf SEO tools don’t gather the proper data. This is a huge mistake, particularly for larger websites.
Mismanaging your website’s crawl can lead to:
This webinar will explore the importance of managing your website’s crawl for increased visibility in search engines.
Google’s Gary Illyes noted there’s no “official” definition of crawl budget in a Google Webmaster blog post:
“Recently, we’ve heard a number of definitions for ‘crawl budget’, however, we don’t have a single term that would describe everything that ‘crawl budget’ stands for externally.”
He goes onto break the budget itself into three components:
The number of simultaneous connections Googlebot may use and also the time it waits between each crawl.
If a site responds quickly to Googlebot, the bot will see the site as healthy and can increase the crawl rate. Conversely, if the site responds poorly, the crawl rate can go down.
If a site doesn’t have much content, or new content being added, or doesn’t have content linked to, the demand for Google to crawl that site reduces.
You can hopefully see there are some things we can have some influence over here and that some of the activity you are doing already has a bigger impact on your ability to rank you than thought.
We can use Google Search Console (or Bing Webmaster Tools) to get the data we need.
Just log in to Search Console > Domain > Crawl > Crawl Stats to see the number of pages crawled per day.
Google doesn’t give any guidance on what these reports mean, so let’s take a look at 3 client websites to give some context:
The website above is experiencing a steady, healthy crawl from the Googlebot. The statistics to the right show the variance depending on daily crawls. These variances are normal, there’s generally a large swing between “high” and “low”.
The website above experience a drastic drop in pages crawled after they misused a directive in their Robots.txt file. The directive told search engines NOT to crawl a large number of pages on their website, causing a sudden drop in pages crawled. In this instance, this was a bad move, as the pages they blocked had value to searchers.
The website above is experiencing an increase in pages crawled per day due to an influx of inbound links from authority websites (causing Googlebot to visit the site more). You could also see an increase in pages crawled by passing equity to pages using internal links OR publishing more content on your website.
Any of the following factors could have a negative impact on your crawl budget.
For larger sites, this is a way to filter and display certain results. While great for users – for Googlebot not so much. This creates lots of duplicate content issues, sends Google the wrong message, and can mean Googlebot crawls pages we don’t want it to.
http://www.example.com/category.php?category=gummy-candies&price=5-10&price=over-10
This can occur pretty easily on custom built eCommerce websites. For example:
This can easily happen on custom built eCommerce websites – the solution is to use canonicalization or dynamic url parameters.
Watch Google’s guidance on configuring URL parameters
Having Google see a 200 (OK) status returned for a page that doesn’t exist and that should be a 404 error. Google doesn’t want to waste time crawling these pages and over time, this can negatively impact how often Google visits your site.
You can check for soft errors in Google Search Console Crawl Errors report.
These happen when links go on and on. Google gives the example of a calendar with a “Next Month” link. Googlebot could theoretically follow those links forever.
Google wants to show users the best search results possible. Having a lot of low quality, thin or spammy content will cause Google to lose trust in your website.
The best way to audit your site crawl budget is to analyze your log files. Your server logs should be readily available but can normally be accessed via cPanel, Plesk or FTP.
Downloading your log files you can use a tool such as botify which can help you understand where Googlebot is spending most of its time on your site.
You are looking to find out:
Once you have identified these pages and errors, we can start to look at increasing our crawl budget.
There are a number of things you can do to increase your crawl budget and are both onsite and offsite elements. Make sure you check all of these methods to increase your crawl budget.
The most common website errors are:
Redirect chains stop the flow of link equity around a site and dramatically reduces Google crawl rates.
Matt Cutts in one of his webmaster help videos back in 2011 recommended 1-2 hops as ideal, three if needed, but suggested that once you got into the 4-6 range, the success rate of Googlebot following those redirect hops was very low.
Make sure you are using your robots.txt file to block parts of your site you don’t need Google to waste its crawl budget on.
These could include:
As Google touched on above, site speed is a factor in your crawl health – the faster your site, the better crawl health you will have and the more Google will crawl your site. You can see that Google has highlighted this as a major reason for reduced crawl budget.
Some of the major elements to investigate will be:
For those with eCommerce platforms that have dynamic URLs, Googlebot treats any dynamic URLs that lead to the same page as separate pages. This means you may be wasting your crawl budget by having Google to crawl the same page over and over as it thinks it is a different page.
Backlinks not only help to improve your organic rankings, they push search crawlers to your site more often.
Why bother?
Based on our data (average of FTF client data)
While this is a part of technical SEO, it is something that gets overlooked by lots of SEOs. Crawl budgets as you can now see are a big part in making sure your site gets the crawl it needs.
If you don’t have the right amount of budget or are using it up sending Google to the wrong pages, your site will never drive you the traffic it should. As you can see from the above, this stuff works. And it can be a great way to unblock a site’s ability to drive organic search traffic.
This site uses Akismet to reduce spam. Learn how your comment data is processed.
See how partnering with us at From the Future can help build your business.
Thanks for this piece Ryan. I love any tip that’ll give me an extra % boost of a leg up on the competition.
You’re welcome!
Thanks for clear lot of issues within one content ! amazing Ryan stewart
If you don’t have the right amount of budget or are using it up sending Google to the wrong pages, your site will never drive you the traffic it should.
Thanks Ryan stewart. amazing share.
I already made a crawl to our site and realize it but it showed occurred and they said you need to fix it.
I’ve heard all about this crawl thing from our website’s seo specialist. Didn’t get it at all until reading this article. Thanks for sharing!
Amazingly, I got a lot of useful information here to optimize my website’s crawl budget. Thank you for sharing this article, it’s easy to read and understand it…
Thanks for the helpful tips Ryan, I’m still getting familiar at managing my own website.
Thanks for laying out all the key points of a crawl budget. The increases in organic traffic and keyword rankings are truly remarkable. Hope to see more material soon.
I like your advice on how to expand your crawl budget by reducing crawl errors and redirect chains. Once you know what to look for and how to utilize all the benefits, a crawl budget is a very useful tool in publicizing content online.
The decrease in your indexation rate and a decrease in your rankings and organic traffic obviously stems from an ineffective management of a site and these two factors should be timely tackled.
Good to know about what crawl budget entails. It can be described as the crawl rate, it’s health and demand.
Hi Ryan a simple idea for discussion:
Could we say “crawl budget” tends to be something reserved for large sites? what about small sites, lets say 150 pages and use no-index tag for lets say 15 pages (10%) Whats the expected ranking uplift .. Can we call that optimising for crawl budget? or 0 to minimal impact? Believe most SEO’s have to work on small sites with a Million things to do, how relevant is in this case.
Hi Hassan, here all I want to say is this is another awesome post by you men. I seriously become your regular reader. You having great expertise and technical knowledge of SEO. I’m doing SEO as well. If I get a chance to work with you I will love to do. As you already have my email you can contact me. I just started an my websites all Pakseos.com. Take a look here and let me know any improvement that you see.
Thanks