Posted on Monday, July 27 2009 by
What a week we had. I was so excited that our new website is attracting more visitors everyday and then suddenly the traffic dropped dramatically from one day to the next. What happened? Well I did a quick search in Google for the term “Life Coach” and our site did not come up in the usual top 3 (when you search on google.com.au). Worst we were not even in the first 50 results and our homepage had dropped completely of the index.
Panic set in as we are an Online Business and depend on people finding us via the popular search engines. I did a quick check on our website in Google’s webmaster tools and found many errors occurred when Google was crawling our site. A little more investigation showed that the robots.txt file was missing and there was no valid sitemap. On top of this due to our new CMS software we had multiple links that pointed to our homepage and many of our old URL’s now also pointed to the homepage.
To make a long story short, Google got very confused and didn’t really know which link belongs to our actual homepage. The result was that the homepage was taken out of the index. Other pages were still there, obviously ranking much lower for our main keywords. Once I knew what was going on there were a couple of steps I took to get our page back in the index.
- Created a new robots.txt file. This file basically tells the search engines, which files to crawl or not to crawl when coming to our site.
- Submitted a new sitemap to the major search engines Google, Yahoo and Bing. This basically tells them which pages are important to you. You should ensure to update the sitemap regularly and resubmit to inform the crawler of any changes to your site. There is a great free tool that creates your sitemap for you.
- Reconfigured a tool for generating URL’s used on our website to ensure I limit the number of URLs that point to the homepage. You can have multiple URLs, that is usually not too much a problem, but you need to ensure there is a proper Redirect implemented.
- Pointed most of our old (“orphaned”) URLs to a new page on our site instead of redirecting to the homepage. Where possible I found the equivalent in the new structure and for the rest I redirected them to a page informing the visitor that we recently moved our website.
Unfortunately it still takes a couple of days until you get back in the good books with Google. We were back in the top 3 within a week. Our next goal is to get a similar ranking globally especially in the US market.
Lessons learnt: Any online business needs to ensure that they regularly monitor their website(s). Use the monitoring tools and reports that are provided to you by the major search engines. This way you know quickly if there is anything wrong and you can act quickly before you see the traffic drop. I was looking mainly at Google Analytics and whilst this is a great tool to know how much traffic you get, where the traffic comes from and what people are doing on your site; it doesn’t allow you to be pre-emptive.
It takes a while to rank high in the index and if something goes wrong you don’t drop out overnight. It is just if you ignore the errors for a while, you will lose your ranking.