Originally Posted by AdultKing

Sphinx won't do the job.

This is clustered nodes of individual components. Crawling is seperate from Indexing. Ranking is seperate from Indexing. Crawls are performed through caching proxies on their own servers.

You could do this with Nutch and ElasticSearch but the overhead would be much greater than this system has.