Quote:
Originally Posted by holograph
we did something like that back in 04-06.
search engine is a huge engineering and expensive project.
AdultKing, even if you get your queries time down to half it is still slow. At some point working our SE project we decided to dump MYSQL and had written out own db engine which were way efficient than mysql.
|
It's terribly expensive. Node is a cluster of 16 nodes at the moment and I'm adding another 16 this week.
The architecture is all NoSQL, the crawler and search engine are written in C and borrow some of the concepts, but not the code, of Lucene. The ranking algorithm is adaptive and reprocesses the index twice a day.
I have development group of servers running where I am tuning the search portion and currently have results within 1.8 seconds max, but I think 1.2 seconds is the sweet spot to make the thing usable.