just an idea how to manage large index that is updated very often.
Very often there is need to update an document in index. To update
document in index you should delete old document from index and then add
new one. In most cases it require you to open IndexReader, delete
document, close IndexReader, create IndexWriter, add document, close
IndexWriter, and re-open IndexSearcher (if index is searched heavily).
Profiling some applications I found that most time is spend in
IndexReader.open() method. Also it produces many objects, so it also
gives GC overhead.
Idea to optimize this process is to create two indexes. One main index
that could be very large and second index that will serve as "change
buffer". We can keep one IndexReader open for the first index. (and use
it for searching and for deleting old documents). Second index is small
and we can reopen IndexReader frequently when needed.
when second index reaches some number of documents we can merge it with
to search this "multi" index we could use MultiSearcher over this two
indexes but with little trick: first IndexSearcher is kept same during
all time till second index is merged with main and second IndexSearcher
is reopened when second index changes.
It is just idea. (It is not tested)
Will it help to improve speed of updating large index and lower memory