Knowing how to submit websites and individual pages to search engines is an essential skill for SEO professionals and webmasters alike.
Whether you’re building a new website or simply adding new content, knowing the ins and outs of indexation is key.
What You’ll Need Before Submitting
First, you’ll need access to edit your website.
Some people may refute this and claim backend web access is not necessary to submit a website to search engines. Well, they’re right.
However, there are some cases where you’ll need access to a website’s backend.
Situations Where You’ll Need Backend Access
- The website doesn’t have a sitemap.
- The website doesn’t have a robot.txt file.
- The website doesn’t have Tag Manager or a way to verify Google Search Console/Bing Webmaster Tools access.
If your client or IT team doesn’t allow you to have access to their backend, or your CMS has certain limitations, see if you’re able to obtain FTP access. This will come in handy later in this article.
Get Access to Google Search Console & Bing Webmaster Tools
If you really want to maximize your organic traffic potential, make sure to submit your website to as many relevant search engines as possible. This may seem pretty obvious, but a little reminder is always nice to have.
So, what will we need?
Most search engines have their own set of webmaster tools to help us manage our web presence. However, the big two you really need are:
- Google Search Console
- Bing Webmaster Tools
Setting up Google Search Console
Before you can submit your website to Google, you’ll need to set up a Search Console account and verify website ownership. You use your Gmail account for this.
If you manage multiple website domains, you will be able to manage all of them from the same account.
Once your account is set up, make sure to follow Google’s guidelines on verifying your website property. There will be prompts that provide you with multiple options to verify your website. If your account is the same as your Analytics account, your website will be auto-verified.
Setting up Bing Webmaster Tools
The first step here is to set up an account. Bing makes this easy by allowing you to sync your existing email accounts to quickly create a Bing Webmaster Tools profile.
Once you’re logged in to Bing’s Webmaster Tools, you’re ready for the next step of submitting your website.
How to Submit an Entire Website
1. Create a Sitemap Index with Categorized Sitemaps
When managing the indexation for an entire website, it’s important to know how to manage it at scale. Having an optimized sitemap can help make this process much easier for webmasters, and most importantly search engines.
When I refer to a sitemap here, I’m referring to an XML file, not an HTML sitemap.
Depending on the nature of your site, it may make sense to create multiple sitemaps to help silo your content into relevant categories. You can even create an image sitemap to help boost your image optimization strategy.
You can use a sitemap index as a root, and link to each sitemap from there.
If you use the Yoast plugin, all of this can be done automatically.
If you aren’t on WordPress, there are many other tools that can help you create your own sitemap. Screaming Frog is my go-to for situations where there’s no automatic sitemap tool.
If you’re more advanced, you can even get fancy with Python to help reorganize XML sitemaps.
Uploading XML Sitemap Through FTP
If you don’t have access to your website’s backend, having FTP access can really save you here.
If this is your first time accessing the backend of your website, this can be tricky. Once you’re connected to your FTP, follow these steps to upload your XML sitemap.
- Search for your public_html directory.
- Open your public_html directory.
- Upload your sitemaps to that directory.
Now you need to test your site to make sure it has been uploaded correctly.
To test, simply copy your file name and add it to the end of your website URL. For example:
2. Optimize Your Robots.txt
So you’ve created and optimized your sitemap. What does a robots.txt file have to do with this?
Well, there are few simple steps to make sure that search engines are able to crawl and index your website.
Add a Link to Your Sitemap
You can actually add a link to your sitemap file in your robot.txt file. This helps search engines quickly locate your sitemap and may improve your crawl rates.
Double Check Disallow Directives
If you’re launching a new website, make sure your website DOES NOT say this:
User Agent: *
That simple two lines of text will block all major search engines from crawling your website. You’d be surprised how often web developers don’t check for this when launching websites.
3. Submit Sitemap to Google & Bing
Remember at the beginning of this article when we set up Search Console and Bing Webmaster Tools? Well, that’s about to come in handy.
Both of these platforms allow us to submit sitemap links. This is a quick way to tell search engines which pages you want them to crawl.
You can submit individual sitemaps, but it may be a bit quicker to just submit your sitemap index.
How to Submit Individual Pages
Did you know that both Google and Bing allow you to submit individual pages?
That’s right! But don’t get too excited.
While some pages can get indexed quickly, sometimes search engines may take longer to index your submitted pages.
Google Search Console – URL Inspection Tool
Google’s new Search Console platform has one of my favorite new tools, the URL Inspection Tool. This is a fairly comprehensive tool that allows webmasters to get instant feedback about how Google perceives certain aspects of a webpage.
One of the best features of this tool is the ability to request indexing. Sound familiar?
That’s because the URL Inspection Tool has replaced the Fetch as Google tool from the old Search Console.
I’ve personally seen instant indexation using this tool, but other SEO professionals have reported slower results.
Bing Webmaster Tools – Submit URLs
In a recent announcement, Bing said that they’re allowing webmasters to submit up to 10,000 URLs per day.
By using the Submit URLs tool in Bing Webmaster tools, you’re helping Bing save crawling resources. Makes sense, but it goes against how search engines typically work.
If Bing is openly encouraging webmasters to submit their URLs for indexation, then why not add this to your marketing checklist?
We covered some direct methods for getting your links indexed, but there are also some indirect methods for getting links indexed.
For those who are new to SEO, it’s important to note that most search engine crawlers discover new webpages through links.
Let’s throw search engines a bone and help them discover our content!
Google Publishers Search API
Google has opened up tools to allow publishers to submit their content directly to Google’s index. This requires some advanced reading, but is well worth the time!
Optimize Internal Links
Optimizing your internal links is a vital part of every essential SEO checklist. Having a structured linking scheme on your site helps search engines discover new pages on your site.
Ramp Up Your Link Building Efforts
Yes, Link building should already be on your SEO checklist.
However, ramping up your link building efforts can help Google find pages on your site. Try earning new links or even reclaim broken backlinks.
The Wrap Up
Some SEO experts may consider implementing all of these tactics to be a bit overkill. However, every little bit helps.
Just like anything in SEO, better performance comes from a culmination of factors, not just one.
How Your Company Can Prevent ADA Website Accessibility Lawsuits
Every day, websites and mobile apps prevent people from using them. Ignoring accessibility is no longer a viable option.
How do you prevent your company from being a target for a website accessibility ADA lawsuit?
Guidelines for websites wanting to be accessible to people with disabilities have existed for nearly two decades thanks to the W3C Web Accessibility Initiative.
A close cousin to usability and user experience design, accessibility improves the overall ease of use for webpages and mobile applications by removing barriers and enabling more people to successfully complete tasks.
We know now that disabilities are only one area that accessibility addresses.
Most companies do not understand how people use their website or mobile app, or how they use their mobile or assistive tech devices to complete tasks.
Even riskier is not knowing about updates in accessibility guidelines and new accessibility laws around the world.
Investing in Website Accessibility Is a Wise Marketing Decision
Internet marketers found themselves taking accessibility seriously when their data indicated poor conversions. They discovered that basic accessibility practices implemented directly into content enhanced organic SEO.
Many marketing agencies include website usability and accessibility reviews as part of their online marketing strategy for clients because a working website performs better and generates more revenue.
Adding an accessibility review to marketing service offerings is a step towards avoiding an ADA lawsuit, which of course, is a financial setback that can destroy web traffic and brand loyalty.
Convincing website owners and companies of the business case for accessibility is difficult. One reason is the cost. Will they see a return on their investment?
I would rather choose to design an accessible website over paying for defense lawyers and losing revenue during remediation work.
Another concern is the lack of skilled developers trained in accessibility. Do they hire someone or train their staff?
Regardless of whether an accessibility specialist is hired or in-house developers are trained in accessibility, the education never ends.
Specialists are always looking for solutions and researching options that meet guidelines. In other words, training never ends.
Many companies lack an understanding of what accessibility is and why it is important. They may not know how or where to find help.
Accessibility advocates are everywhere writing articles, presenting webinars, participating in podcasts, and writing newsletters packed with tips and advice.
ADA lawsuits make the news nearly every day in the U.S. because there are no enforceable regulations for website accessibility. This is not the case for government websites.
Federal websites must adhere to Section 508 by law. State and local websites in the U.S. are required to check with their own state to see what standards are required.
Most will simply follow Section 508 or WCAG2.1 AAA guidelines.
If your website targets customers from around the world, you may need to know the accessibility laws in other countries. The UK and Canada, for example, are starting to enforce accessibility.
In the U.S., there has been no change in the status of ADA website accessibility laws this year.
Some judges have ruled that the lack of regulation or legal standards for website accessibility does not mean that accessibility should be ignored.
Is Your Website At Risk of an ADA Lawsuit?
Some businesses feel as though they are sitting ducks, and rightly so, since in some states, there are individuals and law firms searching for websites that fail accessibility.
Since the Federal government has put a hold on addressing accessibility standards for websites, several states are taking matters into their own hands.
In California’s Riverside County, the DA’s office is pushing back against a law firm and individuals accused of filing more than 100 ADA lawsuits against website owners and small businesses. According to a report by the Orange County Breeze:
“Abusive ADA lawsuit practices are not new, but the defendants in this case are responsible for a significant volume of the ADA lawsuits that have been filed in Southern California over the last several years. Rutherford has been a party-plaintiff in more than 200 separate ADA lawsuits the defendants have filed against businesses in San Diego, Orange, Los Angeles and San Bernardino counties.”
In New York, which saw 1,564 ADA cases in 2018, two plaintiffs filed over 100 ADA lawsuits against art galleries this year. Artnet News reports:
“Technology has changed, that’s why we’re dealing with this, says Frank Imburgio, founder and president of the website development firm Desktop Solutions. “The state of speech recognition and speech synthesis that’s in everyone’s Alexa? That same piece of software embedded in your browser means blind people can avail themselves of your website, but the websites were not designed with that in mind” five or ten years ago.”
New York State Senator Diane Savino, D-Staten Island, chairs the Senate internet and technology committee that considers legislation affecting issues related to technological advancements, like artificial intelligence and digital currency.
It was recently announced they are planning legislative action to curb the surge in the number of lawsuits.
Florida is a hotbed of ADA lawsuits. Flagler County paid over $15,000 to settle an ADA lawsuit brought by a blind person who was unable to use their PDFs.
They removed 7,500 informational PDF documents from their website because they were not optimized for screen readers. In doing so, sighted users no longer had access to this information either.
In the case of Robles v. Domino’s Website and Mobile App accessibility lawsuit, Dominos is taking it to the Supreme Court to fight back.
Every type of website has been the target of an ADA lawsuit. It doesn’t matter if it is owned by one person, a small business or a major corporation.
A Title III public-facing website or mobile app includes travel, hotels, finance, ecommerce, services, healthcare, real estate, and education.
Educational websites and software applications are a growing ADA lawsuit target not only for accessibility to the public, but also for employees and students who use school software.
One recent study found that 95% of U.S. K-12 school websites had errors that made the page difficult for a person with a disability to use. At the state level, schools and universities are facing an avalanche of ADA lawsuits.
What Can Companies Do to Prevent an ADA Lawsuit?
The only way to prevent an ADA lawsuit is to plan for, design and build for web accessibility.
Inclusive design should be a priority and considered the foundation of any website business plan.
Every business with a website, mobile app or internet software application should hire an accessibility specialist who is trained in the application of WCAG guidelines and has knowledge of accessibility laws and guidelines from all countries.
There are only a few companies that specialize in accessibility services, tools or training. They are competitive and busy. You can find alternatives with accessibility consultants focused on just testing or remediation.
If you live outside the USA, you will find accessibility experts and companies who have been doing this work for years and sharing information through podcasts, building new automated testing tools, and stepping forward as accessibility advocates through writings and webinars.
Companies are facing a shortage of accessibility-trained designers and developers. This is a real burden because putting designers on projects who do not know how to build for accessibility is almost as risky as not having anyone at all.
For example, applying ARIA with HTML5 is commonly done incorrectly or image alt attributes are not written properly, especially for infographics or images over background images.
The source of most ADA lawsuits is the inability to access webpages or mobile apps with assistive devices used by sight-impaired users and people who can not use a mouse pointer.
The fun of web design for designers is the visual presentation.
Elementor, a wildly popular WordPress theme building and page design plug-in, makes it easy to incorporate parallax, dynamic content, and animation, can be enhanced to increase the accessibility of the website, and it allows the creation of new themes, headings and footers with more developer control.
What If I Can’t Afford to Hire An Accessibility Specialist?
This question applies to all businesses, but for small and medium-sized businesses, adding an accessibility specialist is out of the question because of budget constraints.
Most small businesses are a team of one person, or the owner has a website person wearing all the hats from SEO to site maintenance, but not accessibility. That is a separate skill.
Find a website company that offers accessibility services. They may provide accessibility testing, accessibility site audits or affordable package deals for their clients such as monthly remediation for PDFs, documents, images, forms, and content spread out over time.
Accessibility reports performed after a company has received a letter of complaint are extremely expensive and unless performed by skilled accessibility specialists, will not hold up in court.
Should I Just Put Up an Accessibility Statement?
The original purpose of an accessibility statement was to show that a site was tested, what standards it meets, what was not tested, and how to contact the company if there are any accessibility issues.
Some accessibility professionals don’t advise using them at all because the pace of technology creates ongoing adjustments to accessibility guidelines.
Unless your company has gone through formal accessibility testing and remediation, I don’t recommend providing an accessibility statement.
Some companies want to put up one that says they are in the process of testing, but users have no proof and there is no accountability here. It won’t hold up in court either.
As a courtesy, every website or application should make it obvious and easy to contact by email or phone, not a form (because they are most often not accessible) and invited to describe the issue they found.
What Can I Do Now to Improve Accessibility?
Optimally, every website or app should not prevent anyone from using it regardless of any physical, mental or emotional impairment, permanent or temporary.
Understanding how to plan, build, and test for accessibility requires advanced knowledge of accessibility to meet Section 508, WCAG2.1 A+ AA guidelines and regulations required by states and countries.
Finding that miracle person who can do all that is unlikely, expensive, and too overwhelming to think about.
Yet, so is an ADA lawsuit, floundering conversion rates, search engine rank roller coaster rides, and a negative brand reputation.
Starting somewhere, here are steps to jump in:
- Do accessibility testing using a free automated accessibility testing tool like WAVE, Axe, or Tenon. You may not understand how to make the repairs, but you will see errors, warnings, and alerts you didn’t know existed.
- Hire an accessibility specialist to perform formal accessibility testing that goes beyond the limitations of automated tools. Some tools are better than others. Some are not kept up to date on standards. No accessibility expert relies on automated tools. They incorporate manual testing, too.
- Ask for a quote for a limited accessibility review or site audit. This is where a sampling of pages are tested rather than every single one.
- Look for agencies that include accessibility design or testing services. They are worth gold for your bottom line.
- Train your web designers and developers. Invest in them. Your online business may depend on their skills. They not only need to know how to code for accessibility but also how to develop the entire methodology for planning, development, testing, and long term maintenance.
- Large corporations should hire accessibility companies that specialize in user testing with disabled users. This is the same as user testing, but with the addition of new personas and real users with various impairments.
- Use your keyboard to navigate your webpage or mobile app. No mouse. If you can’t figure out where you are, where to go or got lost, this is a major issue for accessibility.
- Turn on any screen reader app, download a trial of JAWS, or use your mobile phone accessibility settings, and go to your website or app. You will quickly learn what the experience is like for blind and sight impaired people or multi-taskers who are adapting to the use of audible alternatives for reading.
- If you use any third-party software applications or WordPress plugins, require that it meet accessibility compliance by contract.
- WordPress site owners and designers need to know the basics that can be adjusted from the front-end to improve the accessibility of the site. This includes:
- Font sizes (use em), font faces (use sans-serif)
- Proper heading tags in the right schematic order (H1, H2, H3, not H2, H1, H4, H2)
- Test that colors contrast properly (use any free tool).
- Avoid using color as the only visual indicator that the state of something changed or is an alert.
- Make all PDF’s accessible (Adobe has a tool.)
- Underline text links. If you don’t want every link underlined, create rules in CSS to choose.
- Absolutely no centered text unless it is a heading or sub-heading.
- Describe each image using the alt attribute option. At a minimum, describe what the image is. There are lots of rules for alt text. Start with that one.
- Add the WordPress Accessibility plug-in (see resources below) and use it to add focus states, skip to content and other courtesies. And send Joe a donation for using it. He keeps it updated.
And finally, if you don’t need a CMS website, there is more control over the code if you return to an HTML website.
Fortunately, most of the information is available at free or affordable fees. There are accessibility communities, podcasts, webinars, and the WCAG guidelines themselves.
Microsoft, IBM, Google, and Adobe provide detailed how-to advice.
As you apply inclusive design practices, you will see the benefits for SEO, usability and conversions, brand, reputation, referrals, and customer satisfaction.
Accessibility at its most basic level is a human right. Investing in people is worth it.
In addition to the resources listed in Top 36 Web Accessibility Resources for Digital Marketing Companies, check out:
3 Image Optimization Tips for Faster Speeds & Higher Rankings
Some people are visual learners, some kinesthetic, and others auditory. But I’d argue the majority of us are impatient learners.
In many cases, images are the number one issue affecting website load times.
We all understand that high-quality images contribute to positive user experience, but I wonder have we drank too much of our own Kool-Aid?
A 2016 Google and SOASTA study confirmed that complex webpages with more images had lower conversion rates than webpages with fewer images.
While some websites may benefit being “image-heavy”, we should really look at page speed as a scarce resource. The more we dedicate this resource to loading large, bulky resources, the less effective it will be.
With Google’s redesigned PageSpeed Insights, a lot of people are confused about what it means to serve images in “next-gen formats” and what browser support exists to satisfy Google’s concerns.
Additionally, many people are still relying on old technologies to compress and serve images on their website.
Read on for some pragmatic tips to help you optimize images for faster page speeds while preserving your UX.
1. Use Images Responsibly
Minimalistic design often lends itself to good user experience because it provides greater performance. But you need to ask yourself whether an image is absolutely critical to your design.
For more dynamic webpages, it can be tempting to make images interactive and even a focal point of your design.
Unless you’re a photographer, you can leverage alternative resources instead of images to improve your UX, such as:
- CSS3 for interactive elements.
- Scalable vector graphics for simple logos and designs.
- Embedded videos instead of bulky GIFs.
In fact, vector images tend to maintain greater resolution when being loaded on higher resolution devices.
Furthermore, instead of encoding text within images, which search engines can’t read, you could instead turn to web fonts which improve UX.
Within the PageSpeed Insights tool, you’ll often find that Google recommends deferring images offscreen to improve load outs. This essentially means moving images below the fold so at least the initial loadout for users is faster.
To accomplish this, you’ll need to optimize your critical path rendering.
Inline CSS delivery for header and main document module can also provide users with an instant loadout while all unused CSS and resources are loaded in the backend.
While this is fairly complicated, there are tools to assist in WordPress.
W3 Total Cache caches and minifies virtually every resource in your CDN from HTML all the way to WordPress elements.
This tool also offers HTTP/2 capabilities to further improve loadouts.
2. Automate Image Compression
Probably the two best ways to optimize images for speed are by reducing their size and compressing their files. Image compression is incredibly easy with the right tools at your disposal.
If you’re working with WordPress, you can install Imagify and compress all of your images in bulk in one click. All further images that are added to your library will be compressed using this tool.
This plugin even offers an aggressive setting to compress JPGs and PNGs for massive speed boosts.
If you’re worried about plugins slowing down your site or are working with a more open-source CMS, you can use Image Optim to compress all of your images in a particular folder. It’s drag-and-drop features make it incredibly easy to compress images and then add them to your site.
Additionally, if you’re worried compressing images in your CMS and leaving large file sizes, you can also experiment with compression in the Adobe Suite or Affinity Photo.
In terms of resizing images for different devices, WordPress’s responsive design can do that for you using the ‘srcset’ attribute.
One important thing to keep in mind is that lossy compression may come with the potential to sacrifice the quality of your image.
At larger scale compression, lossy compression will eliminate greater bit depth and provide noticeably reduced file resolution.
3. Serve Images in Next-Gen Formats
Depending on your file format, you will have to use lossy or lossless compression.
Traditionally, we’ve relied on two image formats JPEG (lossy) and PNG (lossless compression).
But tools have begun recommending new image formats, especially for creating fast load speeds and maintaining smooth aspect ratios over mobile devices.
You can choose between a number of new formats, including JPEG 2000, JPEG XR, and WebP.
JPEG 2000 is much like traditional JPEG, but with lossless compression. This means that compressed JPEG 2000 files retain their metadata without much of a loss of quality.
JPEG XR supports both lossless and lossy compression at higher compression ratios.
Unfortunately, both of these formats are unsupported by most major browsers, including Google Chrome and Firefox. This means that you’ll need to have the JPEG or PNG format as a fallback regardless.
The only image format that’s supported by Google Chrome, as well as Firefox is WebP. WebP offers lossless and lossy compression, as well as support for animation.
There are a variety of WebP plugins to experiment with and you can convert a JPEG or PNG file to WebP using an online converter or Photoshop.
Of course, it’s ideal to provide a backup to a WebP file. To accomplish this, you’ll need to leverage the <picture> element in your HTML to provide backups.
Plugins like WebP Express for WordPress automatically do this in the HTML and can be used for bulk conversion in your library.
Ultimately, the option to serve images in next-gen formats is not too appealing considering limited browser support.
But leveraging formats like WebP to give your site images greater compression ratios and smaller file sizes over mobile devices will make a noticeable difference in page speed.
Page speed over mobile is hard to maintain, especially on image heavy sites.
Responsive web design and other speed plugins can’t really account for large bulky file sizes and resources are trying to fetch on your site.
Fortunately, by automating compression, resizing files, and images more strategically you can drastically improve your page load speeds.
Featured Image: Pexels
All screenshots taken by author, April 2019
Google on Content Theft and Effect on Rankings
In a Webmaster Hangout, Google’s John Mueller answered whether plagiarized content could hurt a site’s rankings. Mueller’s response gave a peek into how Google handles sites that steal content and the effect it has on your site.
Scraper Sites and Effect on Ranking
There are many bad actors who steal content and use it on their own sites. It is done with automated software. The process is called content scraping and the sites that publish stolen content are known as content scrapers.
Stolen content is associated with the loss of rankings in Google. It’s not unusual to search a snippet of your own content and see another site ranking with it.
The concern about the effect on rankings is a legitimate one.
Here is the question:
“A few websites have started scraping my content and have been publishing them. We tried to contact their hosts for a DMCA takedown without luck. Does having my content scraped and republished hurt my site? Should I disavow these URLs?”
What is DMCA?
The question made a reference to a DMCA takedown. DMCA is an American law called The Digital Millennium Copyright Act (DMCA).
The law protects hosts, domain name registrars and other businesses from liability for copyright violations as long as they provide a way for content creators to request that stolen content be removed. It also provides due process provisions that allow the takedown to be contested which can then result in costly litigation for the content creator.
It’s somewhat surprising that the publisher tried using the DMCA and failed. This can happen when the web host and/or domain name registrar are in a country outside of the USA. Each country has their own remedy.
Does Copied Content Affect Rankings?
Google’s John Mueller gave an overview of how stolen content affects rankings:
“So from our point of view, other sites copying your content wouldn’t be something that would negatively affect your website. So that’s a very common situation, that sites copy content.
…if you’re not seeing those copies showing up in search for the queries that you care about then it might not be the highest priority to focus on.”
What John Mueller makes sense in the context that scraper sites do not generally rank for actual search queries. Is it possible for scrapers to rank in long tail or non-competitive queries? Almost anything is possible with those kinds of queries.
Why Scrapers Rank for Snippets of Content
It’s not unusual for a scraper site to rank for a snippet of content stolen from another site, but there’s a good reason for that.
Snippets of content are generally regarded as gibberish. If another site ranks for a snippet, it’s not because their thievery has made your site less relevant. It’s because the search algo ranks pages differently for nonsense phrases.
Google’s algorithm is trying to make sense of all search queries. That’s virtually impossible to do when there is no “sense” in the search query.
And when the snippet does make sense, Google may very well rank other sites for that query ahead of your sites, but that’s the algo kicking in, ranking pages for “topics.”
Google does not rank pages by matching keywords, so even if the search is your snippet, that does not guarantee that your site will rank number one.
What’s important is that content thieves generally do not rank for the search queries that matter. So don’t let it trouble you if you see scraper sites outranking you for snippets. That’s not a sign that your site lost ranking strength due to stolen content.
How to Protect Against Scrapers?
WordPress Anti-bot Plugins
There are many WordPress plugins that provide a defense against malicious scrapers.
WordFence is a popular plugin that can be customized to block scrapers for however hours you want to block them. It emails you to let you know when you’re under attack, which can help you increase how swiftly WordFence shuts them out.
WordFence appears to work by monitoring visitor behavior, particularly the amount of pages or the kinds of pages that it is trying to download. It’s the behavior that triggers a wall that blocks the bots.
I use WordFence to stop scrapers and hacker bots and am happy with how it works.
Blackhole Anti-bot WordPress Plugin
Blackhole works on the principle of the honeypot. Good bots will avoid crawling a prohibited link. Bad bots will rush right in. Blackhole sets a trap for bad bots by including a link to the honeypot. Once the bad bot follows the prohibited link the trap is triggered and the bot is excluded from crawling.
All search engines are whitelisted. This means that no legitimate search engine will ever be blocked, even if Google follows the link.
There is a PHP bot blocker called Blackhole. Blackhole can be installed with any server that uses PHP. So it will be compatible with a forum site using software such as Xenforo or phpBB. Read more about the PHP version of Blackhole here.
reCAPTCHA Enterprise Beta
Google recently announced a free beta trial of a service called, reCAPTCHA Enterprise. It is a cloud service that is designed to block automated scrapers, hackers and other malicious bots.
That Google itself is offering a solution to bad bots may be a sign of how important it is to block automated bot software, including scrapers.
Should You Protect Against Scrapers?
I believe it’s a good idea to protect your site from automated bots. Bots tend to crawl at night at the same time that Google and other legitimate bots are crawling. This can become problematic when too many malicious bots are probing your site, slowing down your server. This can cause your server to begin serving error response codes to Google, which will then be unable to crawl your and index your site.
So although John Mueller is correct to say that stolen content does not affect your rankings, you should still try to protect against scrapers in order that Google can properly crawl and index your site.
What’s important is that Google confirmed that scraped content does not affect your rankings.
WordPress2 weeks ago
Google on Content Theft and Effect on Rankings
SEO3 weeks ago
Identity Resolution: Secrets to Success
SEO1 week ago
How to Put Your Rank Tracker Data Into Action
Video Marketing2 weeks ago
The Social Media Video Checklist
SEO3 weeks ago
5 Tools to Track Your Brand Mentions on Social Media
SEO3 weeks ago
Sites vulnerable to XSS can be used to phish Googlebot
SEO1 week ago
Performance Planner now live in Google Ads: Shows predicted impact of campaign changes on conversions, clicks, more
SEO7 days ago
Google Shares Top 3 SEO Factors