Skip to main content

Stop search indexing for Netlify Deploy Previews and Branch Deploys

Posted in Development and Serverless

Now that you’re cool with Netlify Deploy Previews and Branch Deploys, you might be worried that search engines will start indexing them and you end up with duplicate content problems.

Although Deploy Previews and Branch Deploys are unlikely to be linked to publicly, there’s a small chance one might make it out into the wild web; after all, I linked to one in my Deploy Previews article

Here’s how to fix it:

Add a command to your npm scripts

Search engines use a robots.txt file to find out what they should and shouldn’t be indexing. I have an npm script called noIndex:

scripts": {
"build": "this is whatever my build command is",
"noIndex": "npm run build && echo 'User-agent: *\nDisallow: /' > dist/robots.txt"
},

This builds the site as normal, then overwrites the contents of my default production robots.txt file with instructions not to allow any pages to be indexed.

Run the command for Deploy Previews

Then all you have to do is tell Netlify to deploy using that command instead of the default build command you’ll’ve set up for your production site:

[context.deploy-preview]
command = "npm run noIndex"

[context.branch-deploy]
command = "npm run noIndex"

Checking it all works

In the Netlify control panel, head to Deploys and keep an eye on the deploy log for your Deploy Preview or Branch Deploy. As the deployment rus, you should spot something like this if it has worked:

Different build command detected, going to use the one specified in the toml file: ‘npm run noIndex’ versus ‘npm run build’ in the site

Then, if you open your deploy preview or staging site in your browser and navigate to https://deploy-preview-or-staging-subomain.netlify.com/robots.txt, you’ll see it looks like this:

User-agent: *
Disallow: /

So now you don’t have to worry about anything other than your production site being indexed by search engines!

Subscribe

I send a newsletter on the last day of every month, containing:

  • A roundup of the articles I’ve posted
  • A hot pick from my archives
  • Some interesting posts from around the web

I don’t collect any data on when, where or if people open the emails I send them. Your email will only be used to send you newsletters and will never be passed on. You can unsubscribe at any time.

More posts

Here are a couple more posts for you to enjoy. If that’s not enough, have a look at the full list.

  1. XHTML syntax is still worth using

    A few weeks ago I read an article on CSS-Tricks about writing HTML the HTML way, not the XHTML way, and it has been bothering me a bit.

  2. Booleans in ARIA

    HTML booleans are bit quirky but, as if just to complicate things, booleans in ARIA work slightly differently. It’s worth knowing how and why.