I am trying to work on Dataset released by quora, to identify if Question1 has similar intent as of Question2
The dataset looks like:
0|What is the step by step guide to invest in share market in india|What is the step by step guide to invest in share market?|0
I am trying to refer to Abhishek Thakur's feature to get started. It says:
As per my understanding the python code for sklearn would be:
from sklearn.feature_extraction.text import TfidfVectorizer import pandas as pd tfidf_vectorizer = TfidfVectorizer() data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data.question1) data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data.question2)
data['tf_idf_q1] and data['tf_idf_q2] will refer to 2 models for each question as in 1st part of the image.
I am not sure how would i achieve second part? Do I fit_transform the vectorizer with first question and then transform the second question? Or do I merge 2 questions and then get a vectorizer? Something like below:
merged_questions = pd.DataFrame(data['question1'].map(str) + data['question2'].map(str)) data['tf_idf_q1_q2'] = tfidf_vectorizer.fit_transform(merged_questions)
Any inputs are greatly appreciated.