Binary Classification [Text] based on Embedding Distance?

Question

I was just informed this community was a better fit for my SO question. I am wondering if I can use a Milvus or Faiss (L2 or IP or...) to classify documents as similar or not based on distance. I have vectorized text from news articles and stored into Milvus and Faiss to try both out. What I don't want to do is retrain a model every time I add new article embeddings and have to worry about data set balance, do I have to change my LR, etc.

I would like to store embeddings and return the Top1 result for each new article that I'm reading and if the distance is "close" save that new article to Milvus/Faiss else discard. Is that an acceptable approach to binary classification of text from your point of view? If so with DistilBert embeddings, is magnitude (L2) a better measurement or Orientation (IP)?

When I say "close" this isn't a production idea for work just and idea that I can't think through or find other people explaining online, I would expect accuracy of "close" to be some ballpark threshold...

As a Cosine Similarity example (Figure1) if OA and OB exist in Milvus/Faiss DB and I search with a new embedding OC I would get OB closest to OC at 0.86 and if the threshold for keeping is say > 0.51 I would keep the 0C.

As an L2 example (Figure1) if A' and B' exist in my Milvus/Faiss DB and I search for C' with a threshold of say < 10.5 I would reject C' as B' is closest to C' at 20.62.

Figure 1 - medium article

Answer 1

The are two levels to your question:

1. Conceptual - Yes, you can perform an approximate nearest neighbor search on text documents that have been embedded. What you call binary classification is more commonly called anomaly detection when the data is not labeled. Often times in anomaly detection there is a threshold for similar or not.

2. Implementation - Milvus is a database. Faiss is a vector library. The specific implementation will depend on the system is architectured.