Quote:
Originally Posted by goodgirl
Trivial? Not really, "if" it became required to use .XXX and then they blocked other spiders, they then own adult search.
|
Ok, search engines use bots to spider sites. Those bots are in essence programs that visits web pages, save the content in a db, extract the links, visit those pages etc etc.
To visit a page, the bot first needs the ip address of the server on which the content is located. So the bot extracts the hostname from the url and lookups up the IP address.
The domain name system is hierarchical in nature. So let's say gotxxx.xxx belongs to the top-level domain xxx. The ICM Registry controls the records for xxx tld. So in theory it would be possible for the ICM Registry to return different info in reply to nslookups made by computers using IP addresses that the ICM knows are being used by let's say Google than they would in reply to nslookups made by others.
So to 'block' Googlebot effectively the ICM would need to be sure they know every IP address Google uses (and constantly update their database). In theory this would be possible but extremely inefficient.
To circumvent this Google could simply run their queries against nameservers that are also being used by large amounts of surfers. The ICM cannot prevent this, they can't even detect this. And even if they could detect it, blocking thsoe queries would mean that the ICM itself would be preventing large amounts of surfers from visiting .xxx sites.
(for more info on the dns system: rfc1035, rfc1123, rfc2181)
(for more info on the http protocol: rfc1945, rfc2616)