Originally Posted by adultmobile

Well I just loaded the home (or whatever I get after redirects from home are done, if any), this takes a few hours to do 10,000's and I guess days for the million (I kept 16 threads to limit timeouts, could push more for speed). But going to crawl all the pages in the domains (or read from this commoncrawl) gets googleish, unsure what the use for a GFY guy, maybe doing a porn search engine but there's bing videos for this