# How can I preprocess text to feed into a SVM?

I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn.

The code for the above statement

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer=TfidfVectorizer()
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_test['text'])


When I am trying to use the data for an SVM, using a linear kernel and then fitting it into the model using

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, df_test['label'])


I am getting the error

ValueError: Found input variables with inconsistent numbers of samples: [40000, 5000]

df_train is of the shape (40000,2) df_test is of the shape (5000,2)

I am able to overcome this problem by using only 5000 values of df_train using

df_train.loc[:4999,'text']


but this defeats the purpose of having a bigger training dataset.

My question is how can I use the training dataset that will retain it's number of values?

New contributor
JoyBhalla is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

X needs to be the features for your Model and Y needs to be a target variable. As you mentioned, you are using a IMDB dateset so, all the features which you want your model to use will be stored in X variable whereas the 'LABEL' columns will be stored in the Y variable.

X = vectorizer.fit_transform(df_train['text'])