1
$\begingroup$

I have two lists and I want to identify which elements are common (same or similar in meaning or context) in the list. Which NLP algorithm we should use.

list-1= [US, Apple, Trump, Biden, Mango, French, German]

list-2= [State, iphone, ipad, ipod, president, person, Fruit, Language, Country]
$\endgroup$

1 Answer 1

2
$\begingroup$

The most simplest implementation would be using following steps :

Step 1 : Iterate through both the list 
Step 2 : Calculate the Cossine Similarity between each word in list1 with list2
Step 3 : Decide the threshold on cossine similarity. Higher means stricter

Code would be as follows:

list_1 = [ US, Apple, Trump, Biden, Mango, French, German]
list_2 = [State, iphone, ipad, ipod, president, person, Fruit, Language, Country]


# Download the package and model : 
from gensim.models import Word2Vec

similarity_dict = {}
for word_list1 in list_1:
    for word_list2 in list_2:
         model = Word2Vec.load(path/to/your/model)
         cosine_similarity = model.wv.similarity(word_list1, word_list2)
         

Pros :

  1. Easy to implement

  2. Uses Word2vec aso very reliable as word2vec ensure context

  3. Easy to undertsand

Cons :

  1. The code has complexity of O(n*n)

  2. It will not work with Words which are out of vocabulary in word2vec

$\endgroup$
2
  • $\begingroup$ I have already tried cosine similarity. Its not effective, please suggest some other algorithm. $\endgroup$ Feb 9, 2022 at 13:35
  • $\begingroup$ Cosine similarity is just a method to calculate distance.. When you used cosine similarity how did you get vector to compare is what makes all the difference? $\endgroup$ Feb 9, 2022 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.