How can I preprocess text to feed into a SVM?

Question

I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn.

The code for the above statement

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer()
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_test['text'])

When I am trying to use the data for an SVM, using a linear kernel and then fitting it into the model using

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, df_test['label'])

I am getting the error

ValueError: Found input variables with inconsistent numbers of samples: [40000, 5000]

df_train is of the shape (40000,2) df_test is of the shape (5000,2)

I am able to overcome this problem by using only 5000 values of df_train using

df_train.loc[:4999,'text']

but this defeats the purpose of having a bigger training dataset.

My question is how can I use the training dataset that will retain it's number of values?

Answer 1

X needs to be the features for your Model and Y needs to be a target variable. As you mentioned, you are using a IMDB dateset so, all the features which you want your model to use will be stored in X variable whereas the 'LABEL' columns will be stored in the Y variable.

Instead try this code:

X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_train['label'])

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, Y)

Another thing you did wrong here was that, you passed the target variable of you testing data.

You train your model from your training data and if you are confident enough on your model then you test it using the testing data.

Concepts which can help you: Cross-Validation, train_test_split.