Duplicate content: the cancer that stops your organic traffic from growing

Duplicate content is a fairly controversial topic in the SEO world: while Google officials say we should not worry about it, since the search engine can rank web pages perfectly well on its own, most SEO specialists have a panic attack when they discover pages full of duplicate content during an audit. So let’s find out together what duplicate content is, how it affects rankings in Google, what causes it, how it can be removed and what preventive measures you can take.

What is duplicate content?

When the same content, whether an article or a product, can be accessed at different URLs, we are dealing with duplicate content. For example, if the content on the page https://www.admarks.ro/seo/ can also be found on the page https://www.admarks.ro/seo.html, we have two pages with the same content, meaning duplicate content.

We also have duplicate content when someone copies the content from our site onto another site.

How does duplicate content affect rankings in Google?

Duplicate content is not something to neglect at all, because it makes the indexing of a site much harder. When we have several URLs for the same content, Google gets confused and does not know which pages to index, at which point the following can happen:

• Google indexes none of these pages

When Google’s bots crawl a site, certain resources are allocated for this, and those resources are limited (the crawl budget). Put simply, Google does not explore a site endlessly, especially if it detects a lot of pages with irrelevant content, such as duplicate pages, at which point it limits the number of pages that will be indexed.

• Google indexes all the pages

With small sites (up to 10k pages), Google may index every page it comes across, whether or not they are duplicates. As a result, several pages on the same site will compete with each other for the same keywords, a process also known as landing page cannibalization.

• Google indexes only the wrong pages

This is the most frustrating scenario. In practice, it is also possible for the main page, the one we are building backlinks to and hoping to grow in Google with, not to be indexed, with the search engine choosing to index the other duplicate pages instead.

• Google indexes only the right page

In the best case, Google will work out on its own which page should be indexed and ignore the rest. Unfortunately, such cases are quite rare, almost utopian.

The causes of duplicate content

We have established that duplicate content is bad and we have seen how Google treats the problem. Now it is time to establish what causes duplicate content and find out how and why it appears.

• “www” vs “non-www”

A site can work both in the “www” form (for example: www.domeniu.ro) and in the “non-www” form (domeniu.ro). To keep Google from indexing both versions, you need to set a preferred version of the site. For example, we can set www.domeniu.ro as the preferred version, and the domeniu.ro version will be 301-redirected to it.

• “http” vs “https”

If the site uses an SSL certificate, its URLs will no longer start with “http” but will also contain the letter “s” (secured), meaning they will start with “https” (for example: https://www.domeniu.ro). That is why, when the SSL certificate is installed, several procedures are needed, such as redirecting the “http” URLs, updating the sitemaps and setting the preferred URLs in the Google Analytics and Google Search Console accounts, to ensure a migration with no ranking losses.

• Tag pages

Until a few years ago, tag pages were used extremely often, especially on blogs, and an article could end up with several dozen tags. Used excessively, tag pages can do a lot of harm to a site, since a page is created for each tag, containing the articles associated with that tag. So if tags identical to the site’s categories are added, we will have the same content on both the tag page and the category page.

• The mobile version

Some sites that have a dedicated version for mobile devices have the letter “m” in their URL structure (examples: m.domeniu.ro or domeniu.ro/m). As a result, we have two different URLs for each page (the mobile one and the desktop one).

To solve this problem, the mobile URLs need to contain a canonical tag pointing to the corresponding desktop URL. The desktop URL will also contain the rel=alternate tag, pointing to the corresponding mobile page.

• Products that belong to several categories

A common problem for online stores is duplicate product pages. It happens especially when product pages keep the category/subcategory they belong to in their URL structure (e.g. domeniu.ro/categorie/subcategorie/produs) and the same product is added to several categories/subcategories. So if product “abc” is added to both category A and category B, we will have two URLs for the same product (domeniu.ro/categoria-a/produs-abc/ and domeniu.ro/categoria-b/produs-abc/).

• Category pages duplicated by indexed sorting filters

Also on ecommerce sites, when users sort products by price, sales and so on, a specific parameter may be generated automatically at the end of the URL, duplicating the main category page.

• Duplicate content for the homepage

Many platforms automatically generate pages such as domeniu.ro/index.html or domeniu.ro/prima-pagina for the homepage. It is even worse when these pages also appear in the site navigation (breadcrumbs, the home button, the logo, etc.) and build up backlinks.

• Scrapers & plagiarists

When the same content appears on two or more sites, we are talking about external duplicate content. Many online stores run into this problem on product pages, because they often prefer to import the manufacturer’s description rather than create original content for each product.

Unfortunately, plagiarism is also very common online, and sometimes the plagiarists even rank above the sites they copied the content from.

How to identify duplicate content

Identifying duplicate content is not quite as easy as it looks at first glance, and it often takes hours of research to identify the different patterns of duplicate URLs. Fortunately, there are tools we can use to automate the work.

• Google Search Console

Google Search Console can prove extremely useful for identifying duplicate content. To do this, we need to open the “Search Appearance” menu and then “HTML Improvements”. If duplicate meta descriptions or meta titles are reported, we need to download the lists and study them to establish the causes.

• Screaming Frog

Screaming Frog is an essential tool for identifying indexing problems, and it can also help with duplicate content. This tool analyzes all the pages of a site and gives us information about the site structure, meta tags, status codes, hreflang, the types of files on the site and more.

To find duplicate content with Screaming Frog, we need to sort the URLs by the “meta title”, “meta description” or “H1” attributes.

• The “site:” command

Not all the URLs identified by Screaming Frog are actually indexed in Google. To find the indexed duplicate URLs, you can use the commands “site:domeniu.ro inurl:keyword” or “site:domeniu.ro intitle:keyword” when searching in Google.

For example, if the Screaming Frog analysis shows that the filtering system generates the “sort=?” parameter in the URL of category pages, we can then check whether, and how many of, these URLs have been indexed by using the command “site:domeniu.ro inurl:sort=?” in Google.

• Siteliner

Siteliner can help us identify identical blocks of text on different pages of our site, and it also shows the percentage of unique content for each page. This tool can also give us relevant information about the quality of our content.

• Copyscape

Copyscape is one of the most widely used tools for detecting plagiarized content, and it is extremely effective at identifying sites that copy our content.

How to fix duplicate content

Once duplicate content has been identified, it needs to be removed, and we have the following options for that:

• Canonicalizing pages

The handiest way to get rid of duplicate content is to use the rel=”canonical” attribute. This attribute tells search engines which page to index.

For example, if instead of the page “domeniu.ro/produs-sort=?” we want the page “domeniu.ro/produs” to be indexed, we need to define the following canonical tag in the <head> section of the HTML code:

<link rel=”canonical” href=”domeniu.ro/produs” />

• 301 redirects

If the duplicate pages are of no interest to users, they can be 301-redirected to the main pages, and this is also the fastest method. Be careful with the number of redirects, though, as they can slow down the site’s loading speed.

• Blocking content via robots.txt

Robots.txt is a file that tells search engines which pages not to index. This tool is very useful when you want to block URLs containing certain parameters from being indexed.

For example, to block all URLs containing the “sort=?” parameter from being indexed, we need to add the following line of code to the robots.txt file:

Disallow: /*sort=?

• Meta robots noindex

Another very useful meta tag in the fight against duplicate content is “meta robots”. By defining the content=”noindex” attribute in the head section of the page, we tell search engines not to index that page. <meta name=”robots” content=”noindex,follow”>

• Setting the preferred domain in GSC

To make sure our site is not indexed in both the www and the non-www form, we need to set the preferred domain via Google Search Console.

For example, if we want the URLs to be displayed in the www form, we use the “Site Settings” section in GSC and tick the “Display URLs as www.domeniu.ro” box.

• Using clean URLs

An elegant way to solve the problem of duplicate product pages across several categories is to use clean URLs, meaning URLs built directly from the root.

So for a product page, the URL will be “domeniu.ro/produs” instead of “domeniu.ro/categorie/subcategorie/produs”. This way, we can put the same product in several categories without creating duplicate content.

How to prevent duplicate content

The best way to solve the problem of duplicate content is to prevent it from appearing in the first place by taking some precautions.

• Self-canonical

Self-canonical means that a page has the “rel=canonical” attribute pointing to itself. This prevents the indexing of duplicate pages such as those generated by the filtering system, since these dynamically generated pages will automatically have a canonical pointing to the main page.

• Regular checks

Duplicate content needs to be identified early, so it cannot undermine and cannibalize the main pages. That is why it is a good idea to run regular checks with Screaming Frog, with the “site:” search command or via Search Console.