'Search Engines'

Why won’t search engine spiders crawl my site?

23 FEB 2011 2

So you've built up your site, filled it with very good content, built a few quality incoming links and still haven't had a visit from Google? If that's the case, then something may be wrong.

Before we get started, we are talking about your site not being indexed at all (i.e. the query “site:yoursite.com” in Google yields no results) as opposed to not ranking for your designated keywords. Ranking depends on a whole lot of factors and there might be certain keywords where you hardly have a shot at being in the top 10 results in your lifetime.

Misconfigured robots.txt

Some popular CMS’es have a Disallow: * directive in the robots.txt when you first install them, and you will have to switch it off manually. Having a bare-bone site or default installation show up in Google results is equated by some with being caught with your pants down – it’s not wise to allow spiders to crawl your site until you are ready to launch it to the public. Make sure your robots.txt allows crawlers in – this is one of the most common reasons why no spider visits your page.

Banned domain

If you have purchased the domain from an online marketplace (“a man in a white van”) as opposed to having registered it yourself, it could be that the domain has been previously banned by Google. Since the ownership has changed, you can have the name cleared out. Log in to Google Webmasters Tool, authenticate your site and follow the steps to request a reconsideration.

You just need to wait a bit longer

Depending on the amount of incoming links you have built, it can take spiders up to a week to visit your site for the first time. Don’t freak out if your site didn’t get indexed within the first 12 hours. You may start panicking if you don’t see any results in two weeks or more.

Way too many pages

Putting up way too many pages from the very beginning can lead spiders into believing you are a spammer. And it makes sense if you think about it, who builds a site with a million pages and launches it all of a sudden? Pages should be built gradually, more than a few thousand of them overnight is probably too much.

Flash or Javascript links

Even though Google is known to be able to crawl Flash links, don’t rely on it. Make sure you provide a text-only version of your site – at least while it’s still brand new and you need spiders to get a good grasp on what it’s about. Same goes for Javascript-only links, as well as Java or Silverlight plugins.

The above are probably the most common reasons why your website doesn’t get indexed within a reasonable time. Under normal circumstances it shouldn’t take more than a few days until you get at least the front page spidered so wait a little while before you start to worry.

Fill out the form below to get started

find out what we can do for you 877 543 3110