Making the Most of Your Crawl Budget

SEO 3 Comments

Search engines don’t index every page of your site every time they crawl through your pages, unless you have a very small site. The size of your crawl budget is determined by the quality of your site. For Google, PageRank has the most significant effect, followed by the amount of organic search traffic.

The PageRank of your site is, essentially, a quality score assigned by Google. Sites start at 0 and then work their way up as their website grows in authority. PageRank is determined by the number of other pages that link to a page, and the quality of the pages doing the linking. A link from a highly authoritative site will help your page rank more than a link from a brand new site.

The higher your PageRank is, the more time Google will spend indexing your site. A site with a PR value of 1 or 2 will see a smaller number of pages indexed than a site with a PR value of 7 or 8. Because you can’t change your PageRank (at least, not immediately), your goal needs to be to make the most of the crawl budget you currently have.

Knowing that you have a limited budget (especially if you have a very low PageRank), the first thing you need to do is remove duplicate content and fix any 404 (missing page) errors. Don’t let “junk” pages waste the time Google is spending on your site. The second thing you need to do is make every page count. Use keywords and other SEO methods. Offer exceptional content.

Make it easy for Google by providing an XML site map. If you aren’t sure how to do this, there are several online tools that can do it for you, and then explain how to use it (basically, you download the site map from a site map generator, then submit it in Google’s Webmaster Tools section). This provides Google with a map, allowing them to access the different pages of your site in a more efficient manner.

Use a smart linking method. Don’t just provide links back to your home page; link deep into your site to improve the PR value of a more diverse number of pages. Also, learn how to market your pages to build links from outside sources. While building the number of external links you have coming into your site takes time, it will eventually improve your PageRank, and increase your crawl budget.