Quote:
Originally Posted by AdultKing
This is a proof of concept with a fraction of our crawling database, once a few things are ironed out we'll set it up so you can see files as they're crawled and added. We'll eventually run it on our Amazon instances and use proxies to make it harder for UltraMegaBit to block.
|
No matter how sophisticated a crawler is, simple policy methods of anti-tampering, anti-breach, and anti-proxy screening checkpoints could effectively thwart any freewheeling intrusions.