robots.txt - GoFuckYourself.com

OzMan84 · 08-28-2011, 08:16 AM

Hi,

Not sure if ive set this up incorrect but there are still pages getting indexed

Quote:

User-Agent: *
Disallow: /*page/

User-agent: *
Disallow: /*?

domain.com/455/report.php?id=455
domain.com/316/page/popular_likes/0-20/

are two examples of the type pages that are getting indexed

any tips/suggestions would be great

AdultKing · 08-28-2011, 08:36 AM

Globbing and regular expressions are not supported by robots.txt generally

therefore the * has no effect.

OzMan84 · 08-28-2011, 08:44 AM

what would be the wild card entry then? as i have a number of dynamic pages?

AdultKing · 08-28-2011, 08:47 AM

Try reading this page http://tools.seobook.com/robots-txt/

* can be used however you are not using it correctly in your example.

OzMan84 · 08-28-2011, 09:17 AM

ok cheers.

what about a 301 redirect

http://domain.com/1106/
to
http://domain.com/1106/word-word/

it seems to be indexing both versions of the site

Barry-xlovecam · 08-28-2011, 09:46 PM

Quote:

Now this is not quite complete yet, since we need to specifically send a 301 status code. Without the status argument "Redirect" will send a temporary redirect status (302). So we simply take the above example and change it to:

Redirect 301 /foo http://foobar.com/foo

OR

Redirect permanent /foo http://foobar.com/foo

And there is also a specific 'RedirectPermanent' directive as well so you could just as easily say:

RedirectPermanent /foo http://foobar.com/foo

http://www.gnc-web-creations.com/301-redirect.htm

V_RocKs · 08-29-2011, 03:25 AM

Just turn your server off... Nothing gets indexed that way.

Seriously!

Nothing!

08-28-2011, 08:44 AM	#3
OzMan84 Confirmed User Join Date: Nov 2003 Location: Australia Posts: 4,740	what would be the wild card entry then? as i have a number of dynamic pages? __________________ XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information ICQ: 304568058

08-28-2011, 09:17 AM	#5
OzMan84 Confirmed User Join Date: Nov 2003 Location: Australia Posts: 4,740	ok cheers. what about a 301 redirect http://domain.com/1106/ to http://domain.com/1106/word-word/ it seems to be indexing both versions of the site __________________ XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information ICQ: 304568058

08-28-2011, 08:36 AM	#2
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,601	Globbing and regular expressions are not supported by robots.txt generally therefore the * has no effect.

08-28-2011, 08:47 AM	#4
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,601	Try reading this page http://tools.seobook.com/robots-txt/ * can be used however you are not using it correctly in your example.

08-29-2011, 03:25 AM	#7
V_RocKs Damn Right I Kiss Ass! Industry Role: Join Date: Dec 2003 Location: Cowtown, USA Posts: 32,405	Just turn your server off... Nothing gets indexed that way. Seriously! Nothing!