Over the past few months you may have noticed in your account of Google Analytics a unusual increase in traffic from Referral.
But what are referrals? And what kind of referral traffic do you need to watch out for? Before answering these questions, let's recap the main sources of visitor acquisition in Google Analytics.
Sources of traffic acquisition in Google Analytics
If you think about it, the typical sources of traffic capture on Google Analytics are:
- organic traffici.e. visitors coming from search engines, through the keywords for which your site has been ranked on Google by SEO;
- direct trafficincluding visits that arrive on the site by typing the site url into the browser;
- social trafficor people coming from social networks, such as Facebook and Twitter;
- email traffici.e., sessions from links contained in a newsletter sent through a ESP (in this case some ESPs like SendInBlue allow you to add default utm parameters for Google Analytics);
- referral traffici.e. visitors coming to our site from other websites.
We can define Referral as the set of all visits that come to the site from another traffic source, such as links on another site.
So far, so good:
- an increase in organic traffic is a measure of how well we're working from a standpoint SEOperhaps by placing a large amount of long tail keywords;
- an increase in direct traffic can measure a growing degree of loyal users of our site;
- an increase in traffic from social media expresses a growing activity on social networks;
- an increase in traffic from email Finally, it makes us understand that more subscribers to our newsletter open the email and click on the links it contains.
But what does the increase in referral traffic tell us?
In a positive sense that a greater number of sites that have a link to ours (perhaps generated by the activities of Link Building) is bringing us traffic, an implicit indicator of site and link quality.
However, it is important Analyze the sources of referral traffic.
To do so, it is sufficient:
- log into your Google Analytics account;
- Click on "Acquire";
- open the "All traffic" drop-down menu
- click on the "Referral" link.
4 Methods to detect Spam Referrals in Google Analytics
Once you enter the Referral section in Google Analytics, analyzing the sites you may discover visits from suspicious sites, highlighted in red in this screenshot.
There are 4 ways you can help identify these sites:
1. Use OptimizeSmart's custom configuration
The first method consists of download the Google Analytics spam referral report provided by OptimizeSmart, which allows you to create a custom configuration on your own Google Analytics view;
2. Observe the clues
A 2nd method consists in identifying, among the referrals, those with certain clues. Spam referrers usually have 5 concurrent characteristics:
- a % of new sessions of the 100%;
- a bounce rate of 100%;
- a bounce rate of 100%;
- 1 page per session;
- an average session duration of 0 seconds.
What does this mean? That they are always new visits, entering, leaving the same page, staying on the page 0 seconds and not navigating the site.
3. Compare them with the list of spam referrers
You can compare them to the list of spam referrals below.
4. Set custom alerts in Google Analytics
Another method to control the occurrence of spam bots traffic is to set a custom alert in Google Analytics, to let us know as soon as unusual traffic conditions occur, such as a sudden spike in sessions from referrals or an abnormal bounce rate.
To set a custom alert in Google Analytics just go to:
- Administration
- Custom Alerts
But what are these sites?
What is Referrer Spam or Referral Spam
Spam referrers occur when your site receives fake traffic from spam bots and this fake traffic is recorded by Google Analytics.
So the question to ask first is: what are Spam Bots?
Bots are programs usually used for indexing pages online. For example, Google itself has a bot, Google Bot, which it uses to crawl the web and index pages.
These bots are also called crawlers or spiders, hence the spider image that Google is often depicted with:
But bots are not only used for indexing sites, but also for fraudulent uses, such as collecting emails for spam use. In short, there are good bots and bad bots.
Spam Referrers: Ghost Referrals and Semalt-type Crawlers
There are actually 2 different types of referral spam, ghost referrals and Semalt type crawlers.
The Ghost Referrals
The first are the so-called ghost referrals, so called because they do not generate real visits, but ghost precisely, to the website;
Here's how the ghost referrals in an infographic by Ohow:
Semalt-type crawlers
The second are the crawlers used by some monitoring services (such as the famous Semalt).
Depending on the two types, the methods for blocking them are different: if for ghost referrals you can for example use Google Analytics filters, for blocking crawlers like Semalt the methods, as we will see, are basically 4, as the image of MylocalBusiness reminds us:
3 reasons not to ignore the problem
Okay, you'll tell me, but can't I just not care?
You probably wouldn't be reading these lines, but let's talk about the 3 reasons why you should fix it anyway.
- dirty Google Analytics dataThere is no point in monitoring the performance in terms of site visits when the data is tainted by fake visits such as referral spam. Eradicating the problem allows you to regain control of your Analytics data;
- consume bandwidthWhat's the point of consuming bandwidth for automated bots that bring nothing in terms of quality visits?
- create a general deterioration of Google Analytics data: As we have seen before, among the clues that make us discover the spam bot there are elevators bounce rates, time on page null etc.. Certainly not data that give a good image of our site in the eyes of Google.
5 methods to block Spam Referrals permanently
Once we understand what spam referrers are, let's get down to business, and go over the best methods for blocking spam.
We will divide these methods into 5 parts:
- automated softwareThese are software that allow you to automatically block referral spam. It is sufficient to connect, as we will see, your Analytics account and the view, that is the site where we want to make the block of unwanted referrers;
- plugins for WordPress: if you use WordPress such as CMS for your site, there are plugins that can help you in managing referral spam;
- manual method: By creating appropriate filters in Google Analytics you can manually block traffic from spam referrals;
- htaccess: a somewhat technical and delicate method that allows you to solve the problem at its root;
- via php: If your site runs in php you can use a suitable system created by the guys at Nabble.nl.
Blocking Spam Referrers with Google Analytics Filters
Let's start with the manual way to block spam traffic by creating appropriate filters in Google Analytics.
First you have to go up:
- Administration in the navigation menu
- All Filters
- click on + add filter
For the ghost referrals you can create a filter that excludes the hostname:
For spam crawlers, you can use a campaign source filter:
Another possibility is to directly exclude spam referrals like semalt with referral exclusion in Google Analytics:
2 Spam Referrer Software
Referrer Spam Help
Referrer Spam Help is a free service that allows you to identify and filter referral spam.
The service correlates data across many sites and identifies spam based on a list shared across multiple sites and the behavior of referral spam on the sites.
The tool automates the process of identifying spam referrers and creates filters as soon as it detects them.
Referrer Spam Blocker
Referrer spam blocker, similar to the previous service, connects to your Google Analytics account view to create filters against spam bots.
The tool uses the Google Analytics API to automate the process of installing filters on your account.
No technical knowledge required: Referrer spam blocker filters both traffic from ghost referrals and from spam crawlers, installs filters and is free.
Once created, you can find the filters in your Google Analytics admin panel.
Each filter requires multiple API calls and is composed of multiple sources of spam traffic.
WordPress Plugin Against Referral Spam
If you use WordPress, you can use plugins to block unwanted traffic from referrals. Let's look at 4 of them together.
Bot Block (2000+ installations)
The plugin blocks spam referrals saving bandwidth. Referrals are retrieved from a database but there is an option to add them.
The 2 main functions of the plugin include:
- the ability to stop spam traffic before it reaches your site and is counted in Google Analytics statistics;
- save the band.
Among the features:
- Spam traffic can be redirected with a 301 redirect or show a 403 error;
- detailed statistics;
- possibility to add spam subdomains such as spam.semalt.com;
Block Referer Spam (1000+ installations)
Block Referer Spam allows you to block spam bots. It requires no configuration and auto updates the list of spam bots. Features include:
- automatic or manual updating of the spam referrer list;
- possibility of adding new ones;
- block via mod_rewrite or WordPress.
Block Referral Spam (1000+ installations)
Easy to use, no configuration required, the plugin blocks 375+ spam domains.
This plugin blocks spam sites likeemalt.com andbuttons-for-website.com by redirecting spam bots before they reach the site and dirty Google Analytics statistics.
Block Referral Spam via PHP
For those who have a site running in PHP, the guys at Nabble.nl have created a php file package that allows you to block spam crawlers like Semalt: you can find the Semalt Blocker by Nabble here.
Blocking Spam Referrers via Htaccess
Blocking spam crawlers via htaccess
To block spam crawler type referrers like Semalt, via htaccess, you can add the following code:
RewriteEngine On
Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*semalt\.com\ [NC,OR].
RewriteRule .* - [F]
This code will block all http and https referrals from semalt.com and its subdomains.
Block IP address via htaccess
Similarly, you can block the IP address used by the spam bot in htaccess like this:
RewriteEngine On
Options +FollowSymlinks
Order Deny,Allow
Deny from 234.45.12.33
Block range of IP addresses via htaccess
Or, you can directly block the range of IP addresses used by the spam bot as follows: 76.149.24.0/24 is the range in question CIDR, a method used to represent a range of IP address ranges.
This allows you to block more effectively than individual IPs and requires less space on your server.
To convert a CIDR to an IP range and vice versa you can use this tool:
RewriteEngine On
Options +FollowSymlinks
Deny from 76.149.24.0/24
Allow from all
Prevent spam bots user agents from accessing the site
With the following strings you can block spam bots directly:
RewriteEngine On
Options +FollowSymlinks
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC].
RewriteRule .* - [F,L].
You can do the same in Google Analytics by setting up a filter that blocks certain bots and spiders.
An updated list of Referral Spam
You can find numerous up-to-date lists of spam referrers on the net.
Among the various, here are a few:
- https://perishablepress.com/blacklist/ultimate-referrer-blacklist.txt
- https://dl.dropboxusercontent.com/u/12249901/Temp/spam-list.txt
- https://github.com/devimweb/bot-public-list
- https://github.com/piwik/referrer-spam-blacklist
Now it's your turn
How about you? Have you dealt with referral spam? What solutions have you implemented on your site to curb spam traffic from these bots? Let's talk about it in the comments!
