0
$\begingroup$

I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn.

The code for the above statement

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer()
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_test['text'])

When I am trying to use the data for an SVM, using a linear kernel and then fitting it into the model using

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, df_test['label'])

I am getting the error

ValueError: Found input variables with inconsistent numbers of samples: [40000, 5000]

df_train is of the shape (40000,2) df_test is of the shape (5000,2)

I am able to overcome this problem by using only 5000 values of df_train using

df_train.loc[:4999,'text']

but this defeats the purpose of having a bigger training dataset.

My question is how can I use the training dataset that will retain it's number of values?

New contributor
JoyBhalla is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
0
$\begingroup$

X needs to be the features for your Model and Y needs to be a target variable. As you mentioned, you are using a IMDB dateset so, all the features which you want your model to use will be stored in X variable whereas the 'LABEL' columns will be stored in the Y variable.

Instead try this code:

X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_train['label'])

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, Y)

Another thing you did wrong here was that, you passed the target variable of you testing data.

You train your model from your training data and if you are confident enough on your model then you test it using the testing data.

Concepts which can help you: Cross-Validation, train_test_split.

$\endgroup$

Your Answer

JoyBhalla is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.