---
title:
  "Handling domain redirection while maintaining different google crawl
  attributes using Cloudflare"
description:
  "Handling domain redirection while maintaining different google crawl
  attributes using Cloudflare"
canonical_url: "https://www.bigbinary.com/blog/domain-redirection-using-cloudflare"
markdown_url: "https://www.bigbinary.com/blog/domain-redirection-using-cloudflare.md"
---

# Handling domain redirection while maintaining different google crawl attributes using Cloudflare

Handling domain redirection while maintaining different google crawl attributes
using Cloudflare

- Author: Ghouse Mohamed
- Published: August 31, 2023
- Categories: Misc

[neeto](https://neeto.com) is a collection of different software. Each neeto
product gets its own page. For example, NeetoCal gets the URL
https://www.neeto.com/neetocal.

When it comes to actually using these products, you will be redirected to
`https://subdomain.neetocal.com`. Here, "subdomain" would be the subdomain
allocated to you when you signed up for Neeto.

We planned to add "Google sign in" feature to make it easier for folks to both
signup and to log in. During testing, it all worked fine. However, when we asked
"Google" to approve the app "NeetoCal" for "Google sign in" Google demanded that
our users should be able to see the "Privacy Policy" and "Terms of conditions"
on the website. In order to make Google happy, we added a redirection from
"neetocal.com" to "neeto.com/neetocal".

However, Google was not happy with it. The users are logging into
`https://subdomain.neetocal.com` so the "privacy policy" and "terms of service"
should be visible on the domain "neetocal.com" itself.

We are using Cloudflare as our DNS provider. Using the tools provided to us by
Cloudflare we decided to show the content of "neeto.com/neetocal" on
"neetocal.com" without redirecting the user.

Note that in this case, if you type "neetocal.com", then you will see the URL
change to "neetocal.com/neetocal" instantly. That's because the URL of the main
marketing site is "neeto.com/neetocal".

Cloudflare provides
[Page rules](https://developers.Cloudflare.com/support/page-rules/understanding-and-configuring-Cloudflare-page-rules-page-rules-tutorial/)
which we will be using to achieve our goals. Below is a video of how it was
done.

<iframe
  width="100%"
  height="315"
  src="https://www.youtube.com/embed/RodZIBxYBHc"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
  allowfullscreen
></iframe>

## SEO duplicate content issue

The Google search engine doesn't like it when we show exactly the same content
on two different domains. Google thinks that the site is trying to cheat Google
and Google will punish both sites.

We want Google to index our main marketing site https://neeto.com/neetocal and
we want Google to ignore "neetocal.com". One way to tell Google not index the
site is by adding a `noindex`
[meta tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#noindex).

```
<meta name="robots" content="noindex">
```

In the above example, we are asking all bots not to index the page containing
the above meta tag.

We planned to inject this meta tag when a page is rendered for the URL
"neetocal.com" and we will not inject this tag when the page is rendered for the
URL "neeto.com".

Upon more research, we found that search engines also look at the response
headers. Given below is the sequence that search engines follow for indexing the
web pages.

- Crawler gets the raw page source as a response to the HTTP request.
- Crawler checks if `x-robots-tag: noindex, nofollow` header is present in the
  response.
- Crawler checks the meta tags to determine if the page needs to be indexed or
  not.

If a page has `x-robot-tag: noidex, nofollow` then the crawler will not index
the page.

Based on this information, we decided to use
[Response Header Modification Rules](https://developers.Cloudflare.com/rules/transform/response-header-modification/)
feature of Cloudflare. Below is a video of how it was done.

<iframe
  width="100%"
  height="315"
  src="https://www.youtube.com/embed/0AFeM2yyg_A"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
  allowfullscreen
></iframe>

## Links

- [Human page](https://www.bigbinary.com/blog/domain-redirection-using-cloudflare)