An often overlooked place to hide some nitty-gritty search engine optimization is in the robots.txt folder in the root directory of your website. By uploading a simple file, formatted properly, you can tell search engine robots which pages to ignore, which pages to index, and what types of pages they are (mobile or not).
If you’re thorough, you can ensure that all of the great content on your site is indexed once, and only once (to avoid duplicate content penalties), and the mobile content is only served to mobile users (or users who find your site through m.google.com).
The possibility exists that a misuse of, or an outright failure to use, the robots.txt file can cause a site to take a rankings hit. As a result, following is a list of best practices for using a robots.txt file.
Best Practice #1 – Do not use an “allow” line, only use “disallow” to prevent indexing.
Best Practice #2 – Consider disallowing parts of your site that might “break” Google indexing, like areas protected by passwords or dynamically-generated pages. Some great examples of directories to disable indexing for would be /scripts/, /cgi-bin/, /wp-admin/, /cart/, etc.
Best Practice #3 – For WordPress sites, disallow Google’s use of the “?s=” parameter, which will generate search results and index these on Google, rather than the actual pages.
Best Practice #4 – Don’t use comments, as they might confuse some search engine bots.
Best Practice #5 – If you are using “Disallow: /” in your robots.txt file, Google will not crawl your site at all. If you aren’t showing up in the search engines, try removing this line.
Best Practice #6 – Remember that robots.txt is readable by anyone, so if you list directories here that you want to disallow the search engines from accessing, it might clue some users in to where sensitive data is stored on your site. Instead, disallow entire directories instead of individual files, and be sure the directory you are disallowing has a blank index.html file in it so that the directory doesn’t show a list of the files in it when a user manually navigates there.
Image – © Vladislav Kochelaevs – Fotolia.com
I'm the Director of Digital Services and Partner at Ballantine, a family-owned and operated direct mail & digital marketing company based in New Jersey. and started in 1966 by my great uncle!
