Optimizing your Website for Google, Not Just for Search
In a recent post, Beyond SEO: Googlebot Optimization, Neil Patel offers some tips to focus on how Google’s crawlers access your site. This goes beyond SEO – before your site can be successfully SEO optimized, it has to be easy for Google to read it. As Patel notes, “a site’s crawlability is the important first step to ensuring its searchability.”
I highly encourage you to read Patel’s article, but I’m going to summarize some of his major points here. Here are some tips Patel offers for making your site Googlebot-friendly (be warned – this may get a little wonky):
Keep Your Website Simple
Don’t Ignore Your Robots.txt File
Rather than putting instructions for crawlers into the section <meta name=”robots”> on each page of your site, put that content into a robots.txt file.
Patel explains that each website only gets a certain amount of crawl time (or crawl budget) allotted from Google. To make that allotment count, you have to direct Googlebot and tell it where it should and should not spend crawl budget with the robots.txt file. Otherwise, Google will crawl your site on its own and may skip important content that you want indexed.
Patel concludes, “The less Googlebot is spending time on unnecessary sections of your site, the more it can crawl and return the more important sections of your site.”
Keep Your Content Fresh
Patel notes, “Content that is crawled more frequently is more likely to gain more traffic. Although pagerank is probably the determinative factor in crawl frequency, it’s likely that the pagerank becomes less important when compared with freshness factor of similarly ranked pages.”
This is a fancy way of saying that Google likes fresh content. The more recently your site has been updated (and the more regularly you update it), the more often your page will be crawled by Google. The more often your site is crawled by Google, the more potential your site has to improve in search rankings.
“Fresh, consistent content always gains the crawler’s attention, and improves the likelihood of top ranked pages,” Patel notes.
Update your site regularly with fresh content. One good way to do this is through a blog, allowing you to post new content without disrupting the structure of your site.
Linking pages throughout your website helps Googlebots find their way around your site.
Patel offers some advice for a good internal linking structure:
“An accurate way to analyze your internal linking structure is to go to Google Webmaster Tools → Search Traffic → Internal Links. If the pages at the top of the list are strong content pages that you want to be returned in the SERPs, then you’re doing well. These top-linked pages should be your site’s most important pages”
In the example image, I see that most of my main pages are the ones receiving the most internal links. That’s fine, because those are the pages that I want indexed. However, you’ll notice that after my main nav pages, the next page with the most internal links is a staff bio. If I have my choice of pages for Google to index, this one is not at the top of the list. Therefore, I need to adjust my internal links to lessen the profile of that particular page.
Create a Sitemap.xml
A sitemap.xml will provide a site guide for Googlebots to follow. This will ensure that the bots don’t get lost in any parts of the site that they find confusing. It will show the bots the way to the pages that you think are most important.
Formats like WordPress will have plugins to help you create an xml sitemap, or you can use one of the free xml sitemap generators.
Once you’ve created a Sitemap in an accepted format, you can submit it to Google using Google Webmaster Tools. This enables Google to provide you with useful status and statistical information. You can also specify the location of your Sitemap in your robots.txt file.
The First Step In SEO Success
Now that you have made your site attractive to Google, you can start optimizing for search engine success. Follow some SEO basics that I have outlined, and there you go – you are on the road to SEO success.