Stop search indexing for Netlify Deploy Previews and Branch Deploys

Posted in Development and Serverless

Now that you’re cool with Netlify Deploy Previews and Branch Deploys, you might be worried that search engines will start indexing them and you end up with duplicate content problems.

Although Deploy Previews and Branch Deploys are unlikely to be linked to publicly, there’s a small chance one might make it out into the wild web; after all, I linked to one in my Deploy Previews article

Here’s how to fix it:

Add a command to your npm scripts

Search engines use a robots.txt file to find out what they should and shouldn’t be indexing. I have an npm script called noIndex:

scripts": {
"build": "this is whatever my build command is",
"noIndex": "npm run build && echo 'User-agent: *\nDisallow: /' > dist/robots.txt"
},

This builds the site as normal, then overwrites the contents of my default production robots.txt file with instructions not to allow any pages to be indexed.

Run the command for Deploy Previews

Then all you have to do is tell Netlify to deploy using that command instead of the default build command you’ll’ve set up for your production site:

[context.deploy-preview]
command = "npm run noIndex"

[context.branch-deploy]
command = "npm run noIndex"

Checking it all works

In the Netlify control panel, head to Deploys and keep an eye on the deploy log for your Deploy Preview or Branch Deploy. As the deployment rus, you should spot something like this if it has worked:

Different build command detected, going to use the one specified in the toml file: ‘npm run noIndex’ versus ‘npm run build’ in the site

Then, if you open your deploy preview or staging site in your browser and navigate to https://deploy-preview-or-staging-subomain.netlify.com/robots.txt, you’ll see it looks like this:

User-agent: *
Disallow: /

So now you don’t have to worry about anything other than your production site being indexed by search engines!

Get them delivered!

If you enjoyed this and want all the latest articles delivered to your inbox every month, pop your email in the form below.

I don’t collect any data on when, where or if people open the emails I send them. Your email will only be used to send you newsletters and will never be passed on. You can unsubscribe at any time.

More posts

Here are a couple more posts for you to enjoy. If that's not enough, have a look at the full list.

  1. Text snippets on Apple devices

    I’ve been using Apple’s built-in Text Replacement instead of TextExpander for a while now. It’s pretty basic, but it’s free and it does the job.

  2. The right way to use break tags in HTML

    Break tags are often misused. I’ll demo some markup patterns to avoid them, and reveal the one and only legitimate use case I can think of.