I want to make Documents search engine where the user will type a query and top n relevant documents should be shown. I want to use BERT for the searching and the first question is can i use it with an Elastic Database ? Seconed question is which task should i use for the pretrained model 1) Question-Answer 2) Binary Classification as 1 relevant 2 not relevant ?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Train with relevant/non-relevant approach using sentence-transformers. When you train the model you can encode all documents and get their BERT embedding vectors.
Elastic search lets you put these vectors in properties of your corpus, so each document is saved along with its embedding vector.
For each query get the first 1000 candidates and their vectors using elastic search and rerank them in python using cosine similarity and return the results.
Don't be surprised if didnt perform well.
PS: the calculation of cosine similarity can be also done on the fly using Learning-2-Rank plugin of elastic search but I didnt use it myself.
answered Jun 10, 2021 at 8:55
-
$\begingroup$ Thanks for the answer, Shouldn't do the relevance ranking itself with BERT? only the embedding ? $\endgroup$ Jun 10, 2021 at 9:54
-
$\begingroup$ As I wrote you have plugins for that. We did it as mentioned above to test the performance and it was not much better than elastic itself. The point is that more technical your language is, better pure BM25 works. If the language of queries and corpus are fairly natural then BERT might improve upon elastic. Again, you can calculate similarities on the fly using elastic itself elastic.co/blog/… $\endgroup$ Jun 10, 2021 at 11:10
-
$\begingroup$ Okay great so I will try the cosine similarty plugin for elastic itself but if I want to rank using a Bert classifier this has no plugins it will be made separately hence will not take the speed advantage of elastic search right ? @kasra $\endgroup$ Jun 10, 2021 at 14:26
-
$\begingroup$ No ... you can just upload your bert embedding vectors as dense_vector and go with similarity. In case it helped you plz consider accepting/upvoting the answer. $\endgroup$ Jun 10, 2021 at 14:30
-