Remove /sites/ from Your Drupal Robots.txt File Now!

It seems that for years, unknowingly the Drupal community has been effectively blocking themselves automatically from Google's Image Search. I have been wondering why my Fractal art never gets any Image Search traffic, but I figured it was just that I only have a few available online right now. That isn't actually the case. Drupal currently has what I would consider a bug in it's robots.txt, which blocks your sites folder. I couldn't believe it when I heard it, but it's true.

Today, I found an article on Blamcast by John Forsythe that points out the glaring mistake. John noticed something that I noticed on my own site a while ago, but never pursued the issue any further. I had just finished 3 years in the SEO industry and had plenty of work to do at my new job as a Drupal developer. My robots.txt, just like most of the community's was disallowing the /sites/ folder, and even the really smart people that I know who are reading this article. This is a horrible idea by design, as by default Drupal stores all of it's images in the sites folder. This idea gets even worse in a Drupal multisite setup, as multisite's also store their files folder in the sites folder. This is probably why my free files library never gets any hits on the files, as I run all my sites in a large multisite setup. Google shows many types of individual files in it's index, so anyone using a multisite setup is also losing out on that type of search traffic as well.

What you need to do is remove the following line from your robots.txt file in your Drupal root directory.

Disallow: /sites/

That's it!

If you think I am joking, try running a Google site search for your domain and see if any of your uploaded images are listed. If you are using the recommended best practices in Drupal, you probably won't see much.