2
$\begingroup$

I'm going through this guide on semantic similarity and use the code there as is.

I'm applying it to a dataset where each row is typically a paragraph (3-4 sentences, over 100 words). Currently, I have over 100k observations, but this number is likely to grow to 500k.

I want to measure semantic similarity between all rows.

When I test BoW and TFIDF on around 20-30k sample, I don't get any performance issues (even without cleaning, stopwords, etc.).

When I try Word2Vec/Universal Sentence Encoder, however, it takes couple of hours to finish even on 3-4k rows sample .

I also get completely different results, but that's beyond the point.

Is there a way to improve the performance for Word2Vec/Universal Sentence Encoder, especially the latter. (As far as I understand, in Word2Vec, words "good" and "bad" may cancel each other out, which is not good for my speach-like data.)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

One approach would be to profile the code to empirically find the slowest parts. A quick visual scan of the code you referenced relieved inefficiencies.

For example, there are several list comprehensions:

labels = [headline[:20] for headline in headlines]

docs = [nlp(headline) for headline in headlines]

One straightforward way to speed up the code is converting those into generator expressions.

Additionally, there are nested for-loops:

similarity = []
for i in range(len(docs)):
    row = []
    for j in range(len(docs)):
        row.append(docs[i].similarity(docs[j]))
similarity.append(row)

You may not need to do a doc-by-doc comparison.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! Unfortunately, I need the doc-to-doc comparison (1 vs all). From what I gathered, the bottleneck is measuring the similarity (on a sample of 5-10k paragraphs). For example, using a simple example from sbert.net/docs/usage/semantic_textual_similarity.html my code crashes at util.cos_sim(embeddings, embeddings) (CUDA of out memory). I think the only way is to feed embeddings in batches, but not quite sure how yet. $\endgroup$
    – narrativera
    Commented Oct 11, 2022 at 3:34
0
$\begingroup$

Two ideas:

  • I understand that you calculate similarity between every pair of rows/documents, right? If so, the bottleneck is due to the quadratic processing of all the pairs First, you should compare only $(d_1,d_2)$ and not $(d_2,d_1)$ (using indexes: if $i<j$), this saves 50% time. I also assume that the goal is to capture pairs/groups of strongly similar documents. If yes, a method would be to first apply the BoW/TFIDF method (simpler and faster), then to apply the embeddings method only to the pairs which obtain at least some similarity threshold with the first method.

  • A completely different approach: apply topic modelling (LDA, or HDP, or other recent method) on the set of document. This would be likely faster. It might also reveal a different kind of semantic similarity betweem documents.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks, this helps! I understand your point about trying to reduce the dataset to those that show promise, but I think even then it will be a very large dataset. Following your suggestion, I tried mylist = [] for i in range(len(sentence_embeddings) - 1): for j in range(i+1, len(sentence_embeddings)): if i < j: measure=util.cos_sim(sentence_embeddings[i], sentence_embeddings[j]) mylist.append({'index': [i, j], 'score': measure}) else: pass, it's not efficient and I still get a CUDA out of memory error for 10000 sample. Could you suggest an improvement? $\endgroup$
    – narrativera
    Commented Oct 12, 2022 at 3:27
  • $\begingroup$ @narrativera dividing by 2 is useful but not enough for a quadratic explosion, the number of pairs increases too fast. What you could study is the number of pairs above a given threshold $t$ with the TFIDF method. If you plot $t$ as x and number of pairs as y, my guess is that you would find that (1) the number of pairs is very high when $t$ is low and very low when $t$ is high; (2) there's probably a sharp change between these two parts, and a point where $t$ is optimal: catching all/most of the similar pairs while not including too many. $\endgroup$
    – Erwan
    Commented Oct 12, 2022 at 10:53
  • $\begingroup$ thanks! Still, I have almost 500k paragraphs. Even if I select a t that eliminates 99% of all observations, it leaves me with 5k rows (~12.5mn pairs). I'd still have a memory problem. I'm asked to use as many pairs as possible. I was thinking of running the code for 1000 pairs, saving the checkpoint, then exporting the results, cleaning the memory, and restarting the loop. But not quite sure how to code it and whether it's an optimal way. $\endgroup$
    – narrativera
    Commented Oct 13, 2022 at 3:11
  • $\begingroup$ Found this datascience.stackexchange.com/questions/44133/… and this datascience.stackexchange.com/questions/69737/… They refer to FAISS, Dask and Gensim. Not familiar with any of them, will try to figure out. $\endgroup$
    – narrativera
    Commented Oct 13, 2022 at 4:42
  • $\begingroup$ @narrativera 500k*500k/2=125.10^9, it's a lot of comparisons. If you have access to a server or a lot of hardware resources why not, but honestly I would certainly use topic modelling for this: my guess is that it would work better than TFIDF, and possibly even as well or better than the embedding method. And the computing time would be reduced a lot. $\endgroup$
    – Erwan
    Commented Oct 13, 2022 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.