3
$\begingroup$

I have the following code:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import pandas as pd

sentences = ["I have the ability", "I have the weakness", "I have the capability", "I have the power"]

tfidf = TfidfVectorizer(max_features=300)
tfidf.fit(sentences)

X = tfidf.transform(sentences)

k = 2

model = KMeans(n_clusters=k, random_state=1)
model.fit(X)

print(pd.DataFrame(columns=["sentence"], data=sentences).join(pd.DataFrame(columns=["cluster"], data=model.labels_)))

The output looks like this:

index sentence cluster
0 I have the ability 0
1 I have the weakness 0
2 I have the capability 0
3 I have the power 1

As you can see "I have the ability", "I have the weakness", "I have the capability" were grouped in the same cluster (cluster 0) and "I have the power" was grouped into a separate cluster. I think they were grouped randomly and it can't tell which sentences actually mean the same thing. I want a way to be able to group "I have the ability", "I have the capability", and "I have the power" together by specifying that ability, capability and power are synonyms. So basically mapping all words to their synonyms. Is there an existing package for this?

$\endgroup$
1
  • 1
    $\begingroup$ The TFIDF representation is very simplistic, it only gives more weight to rare words. You would need to do something much more complex to represent the semantics of the words, with distributional semantics. $\endgroup$
    – Erwan
    Commented Sep 28, 2022 at 21:40

1 Answer 1

7
$\begingroup$

TfIdf vectors require much more data than that to be useful, but also don't give you the ability to identify synonyms. To do that with vectors and the amount of data you're working with, you'll need a pre-trained vector vocabulary. GloVe vectors are a popular choice to start with, but there will be others you can find and play with that may work better for your explicit purpose.

Note that if you don't limit yourself to vector-based approaches, there are many classical approaches to this problem. WordNet would probably be the first thing I reach for here.

$\endgroup$
2
  • $\begingroup$ I'm open to looking into more approaches but I have no idea where to search for them. Could you maybe list some? $\endgroup$
    – james pow
    Commented Sep 29, 2022 at 15:29
  • $\begingroup$ Added a link to WordNet, which I would use in my first approach. $\endgroup$
    – Andy
    Commented Sep 29, 2022 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.