Quote:
Originally Posted by fris
3 Things about robots.txt I learned today.
you can ignore a certain file type
you can ignore a query string
Code:
Disallow: /*customerID=
hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)
Code:
Disallow: /as
Disallow: /ca
Disallow: /cg
Disallow: /do
Disallow: /la
Disallow: /ma
Disallow: /op
Disallow: /st
|
Yes, you're meant to use it to disallow all the crap on your site.
But I didn't know about the last point. Also, it depends on the spider as to whether or not it listens to every factor in your robots.txt. Some spiders like googlebot are nice, and others are evil
