1
$\begingroup$

I am trying to work on Dataset released by quora, to identify if Question1 has similar intent as of Question2

The dataset looks like:

id|question1|question2|is_duplicate

0|What is the step by step guide to invest in share market in india|What is the step by step guide to invest in share market?|0

I am trying to refer to Abhishek Thakur's feature to get started. It says: enter image description here

As per my understanding the python code for sklearn would be:

    from sklearn.feature_extraction.text import TfidfVectorizer
    import pandas as pd
    tfidf_vectorizer = TfidfVectorizer()
    data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data.question1)
    data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data.question2)

data['tf_idf_q1] and data['tf_idf_q2] will refer to 2 models for each question as in 1st part of the image.

I am not sure how would i achieve second part? Do I fit_transform the vectorizer with first question and then transform the second question? Or do I merge 2 questions and then get a vectorizer? Something like below:

merged_questions = pd.DataFrame(data['question1'].map(str) + data['question2'].map(str))
data['tf_idf_q1_q2'] = tfidf_vectorizer.fit_transform(merged_questions)

Any inputs are greatly appreciated.

$\endgroup$

1 Answer 1

2
$\begingroup$

You can use something like this

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data['q1'])
data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identical'].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)

Combined Model - here you actually learn the transformation for all questions. Then transform each one(question) separately to create features for your model training.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
tf_train_data = pd.concat([data['q1'], data['q2']])
trained_tf_idf_transformer = tfidf_vectorizer.fit_transform(tf_train_data)
data['tf_idf_q1'] = trained_tf_idf_transformer.transform(data['q1'])
data['tf_idf_q2'] = trained_tf_idf_transformer.transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identica'l].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)
$\endgroup$
2
  • $\begingroup$ I don't understand. Where is fit_transform() for the combined model? What we did here is, create a separate frame, convert dataframe to nparray and then train logistic regression. The fitting parameters for data['tf_idf_q1'] and data['tf_idf_q2'] would be different as these 2 are essentially different transformations. I thought the image represented a single transformation for both question sets $\endgroup$ Apr 25, 2017 at 12:44
  • $\begingroup$ Thanks, That's exactly what I was unable to figure out. Thanks a ton $\endgroup$ Apr 25, 2017 at 15:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.