GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - robots.txt (https://gfy.com/showthread.php?t=1035901)

OzMan84

08-28-2011 08:16 AM

robots.txt

Hi,

Not sure if ive set this up incorrect but there are still pages getting indexed

Quote:

User-Agent: *
Disallow: /*page/

User-agent: *
Disallow: /*?

domain.com/455/report.php?id=455
domain.com/316/page/popular_likes/0-20/

are two examples of the type pages that are getting indexed

any tips/suggestions would be great

AdultKing

08-28-2011 08:36 AM

Globbing and regular expressions are not supported by robots.txt generally

therefore the * has no effect.

OzMan84

08-28-2011 08:44 AM

what would be the wild card entry then? as i have a number of dynamic pages?

AdultKing

08-28-2011 08:47 AM

Try reading this page http://tools.seobook.com/robots-txt/

* can be used however you are not using it correctly in your example.

OzMan84

08-28-2011 09:17 AM

ok cheers.

what about a 301 redirect

http://domain.com/1106/
to
http://domain.com/1106/word-word/

it seems to be indexing both versions of the site :(

Barry-xlovecam

08-28-2011 09:46 PM

.htaccess

Quote:

Now this is not quite complete yet, since we need to specifically send a 301 status code. Without the status argument "Redirect" will send a temporary redirect status (302). So we simply take the above example and change it to:

Redirect 301 /foo http://foobar.com/foo

OR

Redirect permanent /foo http://foobar.com/foo

And there is also a specific 'RedirectPermanent' directive as well so you could just as easily say:

RedirectPermanent /foo http://foobar.com/foo

http://www.gnc-web-creations.com/301-redirect.htm

V_RocKs

08-29-2011 03:25 AM

Just turn your server off... Nothing gets indexed that way.

Seriously!

Nothing!

All times are GMT -7. The time now is 02:40 PM.