Prevent Indexing When Building a New Site
July 4, 2008 by Jon
Filed under Everything Else
Whenever you’re building a new site, you always run the risk of having Google (or other search engines) indexing your site before it’s completed. While it’s not a death sentence for your site if this happens, things go so much smoother if you can prevent indexing until your site is ready.
So how do you prevent Google and the other search engines from indexing your site until you’re ready?
Fortunately it’s really easy to accomplish…
Prevent Indexing with WordPress
If you installed WordPress to the root of your domain, simply click on the Settings link inside the WP-Admin area, and then click on the Privacy link. Simply select the following option: “I would like to block search engines, but allow normal visitors”
Now some will tell you that not every search engine will abide by that setting, and they’re right. I could care less though. Google, Yahoo, Ask, MSN/Windows Live, and the other major search engines all abide by the meta tag that WordPress will insert on your pages. That’s all that matters to me.
Prevent Indexing with Robots.txt File
The other way to prevent the major search engines from indexing your new site is to add the proper commands inside your robots.txt file. The syntax is really easy, you simply add the following lines to the file and upload it to the root of your site:
User-agent: *
Disallow: /
Those two commands tell all search engine bots that they are not allowed to spider the entire web site. Since every major search engine’s bots will look for and abide by the directives contained within the robots.txt file, you have effectively prevented them from indexing your site.
Allowing Indexing When the Site is Ready
Eventually you will finish your new site… or at least it will be ready to be indexed. So allowing indexing is really easy. If you use WordPress, simply change the Privacy Settings to the option: “I would like my blog to be visible to everyone, including search engines…” If you used a robots.txt file, simply remove the “Disallow: /” line from the file and re-upload it to the root of your site.
After you have removed the search engine blocks from your site, it’s a good idea to submit your site’s URL or your site’s sitemap URL to the major search engines, so they know to go back and re-index the site again.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Posts
I hope your readers really take notice of this post. I learned this lesson the hard way. I thought I would link to my websites before they were ready to start getting some link love before the content went up.
Long story short, I have had a very hard time putting 5 of those 6 sites anywhere in the rankings because Google thought they were all duplicate content. One of them has now broken out of that mould - Google traffic per day has gone from 0 to 400 in a week - but the rest are still languishing.
Thanks for sharing your experience Jonk.