Once a site is live or has advanced past a certain age, most webmasters don’t really concern themselves with their crawl budget anymore.
As long as you keep linking to new blog posts at some point in your website, it should simply show up in Google or Bing’s index and start ranking.
Only, after time, you notice that your site is starting to lose keyword rankings and none of your new posts are even hitting the top 100 for their target keyword.
It could simply be a result of your site’s technical structure, thin content, or new algorithm changes, but it could also be caused by a very problematic crawl error.
With hundreds of billions of webpages in Google’s index, you need to optimize your crawl budget to stay competitive.
Here are 11 tips and tricks to help optimize your crawl speed and help your webpages rank higher in search.
1. Track Crawl Status with Google Search Console
Errors in your crawl status could be indicative of a deeper issue on your site.
Checking your crawl status every 30-60 days is important to identify potential errors that are impacting your site’s overall marketing performance. It’s literally the first step of SEO; without it, all other efforts are null.
Right there on the sidebar, you’ll be able to check your crawl status under the index tab.
Now, if you want to remove access to a certain webpage, you can tell Search Console directly. This is useful if a page is temporarily redirected or has a 404 error.
A 410 parameter will permanently remove a page from the index, so beware of using the nuclear option.
Common Crawl Errors & Solutions
If your website is unfortunate enough to be experiencing a crawl error, it may require an easy solution or be indicative of a much larger technical problem on your site. The most common crawl errors I see are:
- DNS errors
- Server errors
- Robots.txt errors
- 404 errors
To diagnose some of these errors, you can leverage the Fetch as Google tool to see how Google effectively views your site.
Failure to properly fetch and render a page could be indicative of a deeper DNS error that will need to be resolved by your DNS provider.
Resolving a server error requires diagnosing a specific error that can be referenced in this guide. The most common errors include:
- Connection refused
- Connect failed
- Connect timeout
- No response
Most of the time, a server error is usually temporary, although a persistent problem could require you to contact your hosting provider directly.
Robots.txt errors, on the other hand, could be more problematic for your site. If your robots.txt file is returning a 200 or 404 error, it means search engines are having difficulty retrieving this file.
You could submit a robots.txt sitemap or avoid the protocol altogether, opting to manually noindex pages that could be problematic for your crawl.
Resolving these errors quickly will ensure that all of your target pages are crawled and indexed the next time search engines crawl your site.
2. Create Mobile-Friendly Webpages
With the arrival of the mobile-first index, we must also optimize our pages to display mobile friendly copies on the mobile index.
The good news is that a desktop copy will still be indexed and display under the mobile index if a mobile-friendly copy does not exist. The bad news is that your rankings may suffer as a result.
There are many technical tweaks that can instantly make your website more mobile friendly including:
- Implementing responsive web design.
- Inserting the viewpoint meta tag in content.
- Minifying on-page resources (CSS and JS).
- Tagging pages with the AMP cache.
- Optimizing and compressing images for faster load times.
- Reducing the size of on-page UI elements.
Be sure to test your website on a mobile platform and run it through Google Pagespeed Insights. Page speed is an important ranking factor and can affect the speed to which search engines can crawl your site.
3. Update Content Regularly
Search engines will crawl your site more regularly if you produce new content on a regular basis. This is especially useful for publishers who need new stories published and indexed on a regular basis.
Producing content on a regular basis signals to search engines that your site is constantly improving and publishing new content and therefore needs to be crawled more often to reach its intended audience.
4. Submit a Sitemap to Each Search Engine
One of the best tips for indexation to this day remains submitting a sitemap to Google Search Console and Bing Webmaster Tools.
You can create an XML version using a sitemap generator or manually create one in Google Search Console by tagging the canonical version of each page that contains duplicate content.
5. Optimize Your Interlinking Scheme
Establishing a consistent information architecture is crucial to ensuring that your website is not only properly indexed, but also properly organized.
Creating main service categories where related webpages can sit can further help search engines properly index webpage content under certain categories when intent may not be clear.
6. Deep Link to Isolated Webpages
If a webpage on your site or a subdomain is created in isolation or there is an error preventing it from being crawled, then you can get it indexed by acquiring a link on an external domain. This is an especially useful strategy for promoting new pieces of content on your website and getting it indexed quicker.
Beware of syndicating content to accomplish this as search engines may ignore syndicated pages and it could create duplicate errors if not properly canonicalized.
7. Minify On-Page Resources & Increase Load Times
Forcing search engines to crawl large and unoptimized images will eat up your crawl budget and prevent your site from being indexed as often.
Even certain resources like Flash and CSS can perform poorly over mobile devices and eat up your crawl budget. In a sense, it’s a lose-lose scenario where page speed and crawl budget are sacrificed for obtrusive on-page elements.
Be sure to optimize your webpage for speed, especially over mobile, by minifying on-page resources, such as CSS. You can also enable caching and compression to help spiders crawl your site faster.
8. Fix Pages with Noindex Tags
Over the course of your website’s development, it may make sense to implement a noindex tag on pages that may be duplicated or only meant for users who take a certain action.
Regardless, you can identify web pages with noindex tags that are preventing them from being crawled by using a free online tool like Screaming Frog.
The Yoast plugin for WordPress allows you to easily switch a page from index to noindex. You could also do this manually in the backend of pages on your site.
9. Set a Custom Crawl Rate
In the old version of Google Search Console, you can actually slow or customize the speed of your crawl rates if Google’s spiders are negatively impacting your site.
This also gives your website time to make necessary changes if it is going through a significant redesign or migration.
10. Eliminate Duplicate Content
Having massive amounts of duplicate content can significantly slow down your crawl rate and eat up your crawl budget.
You can eliminate these problems by either blocking these pages from being indexed or placing a canonical tag on the page you wish to be indexed.
Along the same lines, it pays to optimize the meta tags of each individual page to prevent search engines from mistaking similar pages as duplicate content in their crawl.
11. Block Pages You Don’t Want Spiders to Crawl
There may be instances where you want to prevent search engines from crawling a specific page. You can accomplish this by the following methods:
- Placing a noindex tag.
- Placing the URL in a robots.txt file.
- Deleting the page altogether.
This can also help your crawls run more efficiently, instead of forcing search engines to pour through duplicate content.
Chances are, if you are already following SEO best practices, you should have nothing to worry about with your crawl status.
Of course, it never hurts to check your crawl status in Google Search Console and to conduct a regular internal linking audit.
All screenshots taken by author, January 2019
Subscribe to SEJ
Get our daily newsletter from SEJ’s Founder Loren Baker about the latest news in the industry!
WordPress’ Parent Company Acquires Tumblr for Shockingly Low Sum
Automattic Inc., owner of WordPress.com, has acquired Tumblr for what is reported to be very low sum.
The Wall Street Journal reported that Tumblr was acquired for an “undisclosed sum,” however, it was soon revealed the sum was “well below” $10 million.
Dan Primack, business editor at Axios, broke the news about the acquisition price. He tweeted an update after publishing the story, saying the sum is actually below $3 million.
3/ Story updated: Price less than $3 million.
— Dan Primack (@danprimack) August 12, 2019
To put this in perspective, Yahoo acquired Tumblr for $1.1 billion in 2013. Yahoo later wrote down Tumblr’s value by $230 million in 2016 after it failed to generate significant revenue.
In 2017, Verizon gained ownership of Tumblr through its acquisition of Yahoo. Now, Verizon is reportedly selling Tumblr for a fraction of what it was valued at 5 years ago.
Despite what is considered to be a low sum, the acquisition of Tumblr is largest ever for Automattic in terms of price and head count.
As part of the acquisition, Automattic will take on Tumblr’s 200 staffers, so no one will be losing their job.
Another thing that will stay in place is Tumblr’s controversial porn ban. Matt Mullenweg, chief executive of Automattic, tells WSJ: “We’re not going to change any of that.”
Going forward, Mullenweg says that executives will look for ways to share services and functionality between WordPress.com and Tumblr.
In the meantime it sounds as though there will be no immediate changes to either service.
101 Easy (& Cheap) Ways to Drive Traffic to Your Website
In the modern-day landscape of saturated online content, it’s no longer enough to just build your site and wait for people to visit it.
You must be proactive at promoting your site and your brand online.
Admittedly, this is much easier said than done, especially because not everyone has the financial capability to throw into paid ad campaigns and corporate sponsorships.
The good news?
There are several things you can do to promote and drive traffic to your website, all without having to spend hundreds of dollars.
To help you do just that, here’s a list of 101 tactics you can try, grouped by strategy.
Search Engine Optimization (SEO) & Link Building
1. Find the core keywords that match your website’s goals, your industry, and offering.
2. Optimize your website and all of your on-page content for search engines.
3. Focus on Google, but add peripheral search engines like Bing, Yahoo, and DuckDuckGo, to your list.
4. Claim your Google My Business listing.
5. Submit your site to online directories like Yelp, Annie’s List, and TripAdvisor, among others
6. Scour Q&A sites like Quora and look for relevant questions you can create content about.
7. Optimize content with relevant keywords, particularly long-tail keywords signaling intent
8. Get news websites to cover your business and link to your site.
9. Invest time in blogger outreach to see which influencers and industry thought leaders you can reach out and link to your site.
10. Join relevant online discussions on sites like Quora and Reddit.
11. Monitor Google Trends for keyword ideas and trending topics you can write about.
12. Write killer headlines that grab people’s attention and encourage them to click on your article links.
13. Link internally so other pages in your site get attention and a bump in traffic.
14. Make sure your website is optimized for mobile to increase your mobile search rankings.
15. Optimize your site for local search, that means including your city or state in your target keywords.
16. Consider using remarketing on Google Ads for brief periods to drive traffic and sales on your site.
17. Use HARO to look for opportunities to appear on roundups, similar to the one below.
18. Optimize images on your website with alt tags to improve their discoverability on Google Images.
19. Optimize your meta descriptions and title tags so they’re easy to read and aren’t truncated in the search engine results pages.
20. Add your local address to the footer of every page on your site to make sure local searchers find you.
21. Improve your website’s page speeds by following Google’s guidelines and recommendations.
22. Use rich snippets to make your entry on the search engine results pages more clickable.
23. Start a blog if you haven’t already.
24. Create content that’s useful, valuable, and shareable.
25. Create free and paid resources such as case studies, reports, survey findings, etc.
26. Look for guest posting opportunities to get high-authority blogs to link to your site.
27. Create infographics that feature a roundup of industry statistics to increase their likelihood of going viral.
28. Start a regular content series, such as “Did You Know?” or a “Tip of the Day” that your audience can look forward to.
29. Update your blog regularly to get a boost in rankings and traffic.
30. Interview industry leaders and feature the conversation on your blog or YouTube channel.
31. Host a webinar or podcast about topics you’re passionate about and align with your business.
32. Create e-brochures that your audience can share, with links to your site and blog.
33. Invest in video content and upload your videos to YouTube.
34. Write an online/offline column for your local paper, magazine, or community website.
35. Create a press kit you can share with influencers, bloggers, and even other businesses.
36. Comment on other blogs relevant to your industry.
37. Launch a free ebook to generate interest in your brand. Offer it as a free download for users who sign up for your newsletter.
38. Start a blog on Tumblr. This is a great content platform, especially if you have a young audience.
39. Have a healthy mix of evergreen content and trending content to increase your website’s discoverability, particularly on search engines.
40. Promote your content on social media channels.
41. Obviously, you want to go big on Facebook. Create a page there if you haven’t already.
42. Join discussions on Facebook Groups to generate visibility.
43. Leverage social media contests on Facebook, Twitter, and Instagram to drive traffic to your website.
44. If it’s relevant to your audience, create an online presence on Snapchat and Pinterest.
45. Use Pinterest to upload high-quality images of your products.
46. Use relevant trending hashtags on Twitter to drive users to your site.
47. Try promoted tweets to fast-track traffic to your site.
48. Start an Instagram account and make sure your bio is filled features your website URL.
49. Use Facebook and Instagram Stories to engage your audience and raise brand awareness.
50. Let your employees control your Stories for a day. This will encourage them to share your social media account (and website) with their personal network.
51. Start an official YouTube channel. Use it to share videos of your brand, your products, and services.
52. Use Facebook, Twitter, and Instagram ads at short bursts to boost traffic to your site.
53. Leave comments on other social media pages.
54. Link your official social media channels with one another. Make sure all pages/profiles have a URL to your website.
55. Take advantage of Facebook and Instagram’s live streaming platforms. This will give you a chance to show your brand’s personality and encourage website visits.
56. If you serve a B2B market, double down on LinkedIn. According to a report, 63% of marketers rated LinkedIn as the most effective B2B social media platform.
57. Use SlideShare to create your own high-quality slideshows. Optimize your slideshow for keywords and add your website URL to the final or beginning slide.
58. Participate in events and talk about the event experience in your blog.
59. Collaborate with local academic institutions to get your brand’s name out in the world of academia.
60. Look for public speaking engagements in your industry.
61. Add QR codes to your print collateral (e.g., posters, post cards, flyers) to drive people to your site.
62. Design beautiful business cards and add a QR code directing people to your site, lead form, or social platforms.
63. Support local organizations to ensure your community knows your brand and site.
64. Place stickers and/or decals on your personal or company cars to promote your website.
65. If you have the budget, pay for local ad placements in your newspaper, benches, sporting events.
66. Organize events such as concerts, poetry nights, garage sales, flea markets, and workshops.
67. Make sure your website URL is visible on company merchandise.
68. Send direct mail and place your URL on letters.
69. Include your website URL on company uniforms
70. Look for free press release opportunities on magazines and newspapers
71. Add your website URL to office signs.
72. Join networking events in your city or out of state.
73. Take advantage of classified ads in your local paper.
74. Support a local charity by sponsoring a fun run or donating part of your proceeds to a cause.
75. Contact your local news station to submit yourself as an expert in your field or industry resource.
76. Join your local Chamber of Commerce or other business groups.
77. Appear on a local radio program as a resource guest, which will let you promote your site as well.
78. Entice customers with an exclusive deal that can be redeemed on your store. According to one study, 57% of shoppers are motivated by coupons to make first-time purchases.
79. Offer free gifts to in-store customers and add material to promote your site.
80. Start a loyalty program requiring users to fill out a form on your website.
81. Offer freebies that can be redeemed on your site after shoppers make in-store purchases.
82. Start a referral network and encourage users to refer your website to their friends in exchange for discounts/deals.
83. Send thank you cards or emails to your in-store customers, placing a URL to your site.
84. Take advantage of seasonal offers (e.g., Christmas, Thanksgiving, Halloween) to increase your likelihood of being found during these occasions.
85. Remember to market your website across all online channels you own — social media, newsletter, blog.
86. Create an official email signature with your website in it (and most recent blog post if applicable).
87. Don’t neglect email marketing. Start a newsletter and incentivize signups with offers and discounts.
88. Encourage customers to leave reviews on your social media pages and website.
89. Take advantage of user-generated content through contests or competitions. Have users submit entries to your website.
90. If your website has been around for a while, consider a redesign to drum up interest when relaunching it.
91. Add social buttons to your blog content and landing pages to make sharing easy.
92. When looking for influencers, look for those who are relevant to your brand and have an engaged audience. The number of followers isn’t a reliable metric for an influencer’s influence.
93. Educate your audience instead of selling to them.
94. Talk and listen to your customers about what they want from your brand. Use this information to improve products and/or create content.
95. Sell yourself and your site wherever you go. You are your greatest ambassador.
96. Use strong calls-to-action in your social media posts and blog content to drive audiences to your website.
97. Look for opportunities to appear on other people’s podcasts or webinars.
98. Make sure your website looks great on all devices to maximize its discoverability.
99. Make sure your internal stakeholders are encouraged to spread the word about your brand and website.
100. Pay attention to what your competitors are doing on their website and do something they aren’t so you stand out.
101. Want visits? Ask for them online and offline from the people you meet every day.
Over to You
This list of tactics only scratches the surface of what you can do to promote your website.
Nevertheless, these tricks should get you off to a good start.
Screenshots taken by author, August 2019
In-post Image #1: HubSpot
In-post Image #2: Content Marketing Institute
10 Best Readability Tools to Check Your SEO Content
Content strategy is a science and every detail matters.
Those details include:
- The reading level of the content.
- Word count (especially in relation to top-ranking content).
- Format and correct use of subheaders.
- Keyword distribution.
- Use of phrases related to the keyword.
- Sentence structure.
In fact, details like these can mean the difference between a No. 1 ranking (or answer box!) vs. content that doesn’t even make it on Page 1 of the search results.
Competitive content writers will use every tool at their disposal as they create content.
That’s why we’ve rounded up the 10 best content writing tools for SEO that specifically help you improve readability.
Add your keyword phrase (and related keywords if you like), and the SEO Writing Assistant will give you an aggregate score based on factors including:
- Number of hard-to-read sentences
- Long words
- Word count and reading time (compared to top-ranking content)
- Tone of voice
Notably, with the SEO Writing Assistant is the one tool on this list where you can set the preferred readability level you’d like your content to have.
You can customize the tone of voice you’d like your content to have, ranging from casual to formal, and check whether any content in the document is plagiarized.
It will also show related questions you should consider posing/answering within the content.
Personally, my favorite aspect of the SEO Writing Assistant is the recommended keywords – the tool will automatically show about 20 phrases that are present in the top-ranking comment.
Yoast is a free WordPress plugin that many digital marketers use to check the basic SEO of their content, but it can also give you a content readability score.
Within the content readability score, you’ll find a report that breaks down:
- Flesch reading ease
- Use of passive voice vs. active voice
- Subheading distribution
- Variety of sentence structure
- Paragraph length
- Sentence length
- Use of transition words
Like most of the tools on this list, the Content Experience provides scores pertaining to your content’s word count, sentence structure, keyword coverage, phrase repetition, etc. based on your keyword target.
This is one of the more robust content marketing tools on this list, and provides insights into user intent, keyword selection, and even the best time of year to publish a piece of content.
As the name suggests, the Readability Tool focuses primarily on the readability of your content.
You can input content you’re working on directly into the tool, or you can use a URL for content that already exits (yours or your competitors).
The report will give not one but six readability scores, including:
- Flesch reading ease.
- Flesch-Kincaid grade level.
- Gunning Fog Score.
- SMOG Index.
- Coleman-Liau Index score.
- The Automated Readability Index score.
It will also show you:
- The number of sentences
- Number of words
- Number of complex words
- Percent of complex words
- Average words per sentence
- Average syllables per word
With the Text Optimizer, you can pop in a webpage and this content readability tool will check the health of your content.
If you’re new to SEO or content strategy, this is a good tool to start with, given that no technical knowledge is required to wield this tool and create great content.
In addition to assessing your content’s word count, sentence length and verb use, this tool will give you suggestions of words to add to your content and words to remove from your content to increase your potential to rank.
According to Text Optimizer, 70% of their users achieve better SEO rankings within five weeks after using the tool.
This content readability tool focuses solely on reading level and gives your content a readability score based on
Not only will it give your entire content a score, but it will also score your content’s individual content.
If writing is not your strong suit, Grammarly is a game-changer.
This tool focuses on the mechanics of writing rather than the science of SEO content – nonetheless, it’s incredibly valuable and belongs on this list.
Poorly written content equates to poor user experience (and high-quality content should be written with the search engines and the user in mind).
Grammarly will address issues in grammar and spelling, but also in tone and structure.
It will flag overly complex sentences and keep an eye out for clarity and conciseness.
You can also set a goal for the content your writing so Grammarly can tailor its recommendations to your project.
Goals are aligned by the type of your content.
You can choose from academic, business, technical, creative, and casual. For most web content, you’ll probably want to choose business.
When working on a piece of business content, Grammarly will flag any use of the passive voice and misuse of pronouns, but allow for some use of informality.
Not only can you use Grammarly for blog posts, site content, and articles, but you can also use it for emails, messages, and social media posts.
Similarly, the free Hemingway app helps you improve the mechanics of your writing.
Copy your content into Hemingway’s desktop app and it will show:
- The readability of your content by grade level
- Opportunities to use more concise language
- Overuse of adverbs
- Use of passive voice
- Sentences that are hard to read
- Places where a simpler phrase could be used
- Word count and character count
- The average length of time to read the content
Sometimes content readability isn’t an issue of keyword distribution or poor grammar.
Sometimes, you just may have wild, out-of-hand formatting that needs to be dealt with.
As the name suggests, Bulk SEO Tools will help you take care of formatting issues (that impact readability) in bulk.
Let’s say, for example, your entire text or large portions of your text are in all uppercase.
You can input that text into Bulk SEO Tool’s case converter and switch the case to sentence case, capitalized case, lower case, title case, etc.
Bulk SEO Tools also has text tools to quickly remove any duplicate lines, add or remove line breaks, and even add prefixes or suffixes if you’re working with a list.
You know what else can hurt readability (at least from a user perspective)?
No one likes a cliché – they’ll make your writing seem contrived and contrite.
This free tool will highlight any clichés in your text in red so you can swap that out for a sentence that’s more meaningful.
SEO7 days ago
Don’t pay more later. Register for SMX East now.
SEO2 weeks ago
Review counts matter more to local business revenue than star ratings, according to study
SEO2 weeks ago
Here’s how Google Ads’ new keyword selection preferences work
WordPress2 weeks ago
10 Best Readability Tools to Check Your SEO Content
SEO3 weeks ago
More Automated Intent Classification Using Deep Learning
SEO3 weeks ago
Google Search Console doesn’t show data for featured snippets read by Google Assistant
Video Marketing2 weeks ago
The First Episode of “Brandwagon” Is Here!
SEO2 days ago
7 Expert Tips to Boost Your PPC Performance Today