I am using an IMDB dataset which contains reviews of the movies in the column text and the rating 0 or 1 in the column label. I am preprocessing the text using Tfidf using sklearn.
The code for the above statement
from sklearn.feature_extraction.text import TfidfVectorizer vectorizer=TfidfVectorizer() X = vectorizer.fit_transform(df_train['text']) Y = vectorizer.transform(df_test['text'])
When I am trying to use the data for an SVM, using a linear kernel and then fitting it into the model using
classifier_linear = svm.SVC(kernel='linear') classifier_linear.fit(X, df_test['label'])
I am getting the error
ValueError: Found input variables with inconsistent numbers of samples: [40000, 5000]
df_train is of the shape (40000,2) df_test is of the shape (5000,2)
I am able to overcome this problem by using only 5000 values of df_train using
but this defeats the purpose of having a bigger training dataset.
My question is how can I use the training dataset that will retain it's number of values?