Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Q1: how to feed 2 input to LR and SVM?
My dataset consist of three columns which are: sentence1 , sentence 2, and label (1 if the sentence2 is a paraphrased of sentence1)
I prepare my data and convert it numeric features using (tf-idf) now I would like to train a classifier, but all the tutorials I find do one input and one output while I would like two inputs and one output. Could you help with an example?
Picture of my data:
The simplest option in order to represent the two sentences independently of each other is to represent each of the two sentences with its own TFIDF vector of features and concatenate the two vectors. In other words you obtain 2 * N features where N is the size of the vocabulary.
But at first sight it looks like the wrong approach for the problem that you're trying to solve: LR or SVM are unlikely to capture the high-level nature of paraphrasing, especially if fed with only basic vocabulary features like this. A slightly more advanced approach would be to provide the model with features which represent the relationship between the two sentences: length, words in common, readability measure, etc.
You can calculate the text-similarity using Transformers. With transformers, we can get better accuracies. Try the following code:
pip install sentence-transformers==1.2.1
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('distilbert-base-uncased')
sen = [sentence1 , sentence 2]
sen_embeddings = model.encode(sen)
from sklearn.metrics.pairwise import cosine_similarity
#let's calculate cosine similarity for sentence 0:
cosine_similarity(
[sen_embeddings[0]],
sen_embeddings[1:]
)
If the similarity score is greater than 0.6 ( or 0.7), you can assume that sentence2 is a paraphrased of sentence1.
