View Single Post
Old 07-27-2011, 06:00 PM  
PornStarToys
Confirmed User
 
PornStarToys's Avatar
 
Industry Role:
Join Date: Jul 2011
Location: A Whale's Vagina
Posts: 581
Quote:
Originally Posted by fris View Post
3 Things about robots.txt I learned today.

you can ignore a certain file type

Code:
Disallow: /*.avi$
you can ignore a query string

Code:
Disallow: /*customerID=
hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)

Code:
Disallow: /as
Disallow: /ca
Disallow: /cg
Disallow: /do
Disallow: /la
Disallow: /ma
Disallow: /op
Disallow: /st
Yes, you're meant to use it to disallow all the crap on your site.

But I didn't know about the last point. Also, it depends on the spider as to whether or not it listens to every factor in your robots.txt. Some spiders like googlebot are nice, and others are evil
__________________
Porn Star Toys - Sex Toys Molded from Porn Stars!

Sex Toy Craze
PornStarToys is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote