0
$\begingroup$

I have a dataframe that looks like this:

sentence    intent
hi          greeting
hello       greeting
buy this    buy
whats up    conversation
.
.

What I'd like to do is take this dataframe, compute the TF-IDF and then use the TF-IDF values on a new query to calculate cosine similarity. For example, if the user types in "hi how are you?" the sentence that is most similar will be printed out with its intent.

Currently I have the TF-IDF of the dataframe:

from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()
x = v.fit_transform(intent_data["sentence"])

How can I get the TF-IDF of the new sentence and then use it to get cosine similarity to find the document (sentence) most closely related to the user typed sentence? Note: I know the dataframe example has very short sentences, but it is just used as an example.

$\endgroup$
0
$\begingroup$

Instead of performing fit_transform. You can just fit the vectorizer on you training data and transform the given query sentence to get it's vector, you can then find it's cosine similarity. You can also then perform inverse_transform to return the terms. Refer sklearn tfidf documentation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.