1
$\begingroup$

I am trying to work on Dataset released by quora, to identify if Question1 has similar intent as of Question2

The dataset looks like:

id|question1|question2|is_duplicate

0|What is the step by step guide to invest in share market in india|What is the step by step guide to invest in share market?|0

I am trying to refer to Abhishek Thakur's feature to get started. It says: enter image description here

As per my understanding the python code for sklearn would be:

    from sklearn.feature_extraction.text import TfidfVectorizer
    import pandas as pd
    tfidf_vectorizer = TfidfVectorizer()
    data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data.question1)
    data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data.question2)

data['tf_idf_q1] and data['tf_idf_q2] will refer to 2 models for each question as in 1st part of the image.

I am not sure how would i achieve second part? Do I fit_transform the vectorizer with first question and then transform the second question? Or do I merge 2 questions and then get a vectorizer? Something like below:

merged_questions = pd.DataFrame(data['question1'].map(str) + data['question2'].map(str))
data['tf_idf_q1_q2'] = tfidf_vectorizer.fit_transform(merged_questions)

Any inputs are greatly appreciated.

$\endgroup$
2
$\begingroup$

You can use something like this

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data['q1'])
data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identical'].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)

Combined Model - here you actually learn the transformation for all questions. Then transform each one(question) separately to create features for your model training.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
tf_train_data = pd.concat([data['q1'], data['q2']])
trained_tf_idf_transformer = tfidf_vectorizer.fit_transform(tf_train_data)
data['tf_idf_q1'] = trained_tf_idf_transformer.transform(data['q1'])
data['tf_idf_q2'] = trained_tf_idf_transformer.transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identica'l].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)
| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't understand. Where is fit_transform() for the combined model? What we did here is, create a separate frame, convert dataframe to nparray and then train logistic regression. The fitting parameters for data['tf_idf_q1'] and data['tf_idf_q2'] would be different as these 2 are essentially different transformations. I thought the image represented a single transformation for both question sets $\endgroup$ – Ronak Agrawal Apr 25 '17 at 12:44
  • $\begingroup$ Thanks, That's exactly what I was unable to figure out. Thanks a ton $\endgroup$ – Ronak Agrawal Apr 25 '17 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.