Machine learning - Algorithm suggestion for my problem using NLP

Question

I am looking for a machine learning algorithm for my problem.

I have a set of sentences like,

["The cat in the hat disabled", "A cat is a fine pet ponies.", "Dogs and cats make good pets.","I haven't got a hat."]

and the search-words like,

["cat","hat"]

I want to convert my sentence list and search-words to a vector space and find matching score between my sentence list and search-word list.

the type of output I am expecting is,

[("The cat in the hat disabled",0.9), ("A cat is a fine pet ponies.",0.5), "(Dogs and cats make good pets.",0.6),("I haven't got a hat.",0.49)]

Please suggest a machine learning algorithm for my task, if possible please share a reference link.

let me know if you have any queries or need more information. I am currently using semanticpy for this https://github.com/josephwilk/semanticpy

I am getting key-error for many search-words. its not performing stemming and lemmatization for the sentence list but only performing for the search-words list.

Please help on this.

Answer 1

There is a really good video about this topic from a PyCon in 2016. There is a pretty in-depth description on how to vectorize your sentences as well as make predictions based on those vectors.

I think that would greatly help you out. That is what I used when I was learning about how to perform sentiment analysis.

Answer 2

This looks like a proper task for Doc2Vec, which is an algorithm to build paragraph embeddings. For a nice implementation with usage examples you can try out gensim.

Other options can be using word2vec and using vector averaging or sum to build sentence vectors (look at this).

For more approaches have a look at these two tutorials where you can see how to implement LSA, LDA, TFIDF:

https://nlpforhackers.io/topic-modeling/

https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730

Answer 3

You need to do fit_transform first then transform, Here sample example

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer


train_set = ["president of India","machine learning is awesome", "python is awesome", "thanks for reading"]

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix_train = tfidf_vectorizer.fit_transform(train_set)
tfidf_matrix_test = tfidf_vectorizer.transform(["president"])

print(cosine_similarity(tfidf_matrix_train,tfidf_matrix_test))