Quote:
Originally Posted by mortenb
I'd look into using stormcrawler.net or Apache Nutch. Stormcrawler seems to be faster ATM, but YMMV. Should be relatively easy to set up. You will still need some beefy hardware and a good connection though.
|
thanks., found out what was casuing the slow speed., bottleneck was innodb and threads count.,
now 250 threads
and batch inserts
100k completes in 23 minutes.,