I'm building an internal semantic search engine using BERT/SBERT + ElasticSearch 8 where answers are retrieved based on their cosine similarity with a query.
The documents to be searched are somewhat domain-specific, off the top of my head estimation is that about 10% of the vocabulary is not present in Wiki or Common Crawl datasets on which BERT models were trained. These are basically "made-up" words - niche product and brand names.
So my question is:
- Should I pre-train a BERT/SBERT model first on my specific corpus to learn the embeddings for these words using MLM?
- Can I skip pre-training and start fine-tuning a selected model for Q/A using SQUAD, synthetic Q/A based on my corpus and actual logged user queries?
My concern is that if I skip #1 then a model would not know the embeddings for some of the "made up" words, replace them with "unknown" token and this might lead to worse search performance.