I was building a text classifier which takes into account certain features of the text and classifies them into two - "Yes" or "No".

I have trimmed the text, removed stopwords and have applied TFIDF on it and now I'm performing SVM classification on it and getting around 81% accuracy on a dataset of size 25,000 with 80:20 training and test split.

I'd like to include some features in the text and have my classifier predict on that.

Example of how my data might look like:-

enter image description here

The code I was using for predicting the label just using text is:-


review_train, review_test, label_train, label_test = train_test_split(meatydf['text'],labeldf, test_size=0.2,random_state=13, shuffle=True)

X_train_tf = tf_vect.fit_transform(review_train)
X_test_tf = tf_vect.transform(review_test)

clf = svm.SVC(C=10,gamma=0.001,kernel='linear')
pred = clf.predict(X_test_tf)

I want to put all the columns of meatydf(text, feature1, feature2 and feature3) as features to help the classifier predict the labels and so I passed the entire dataframe(meatydf) to train_test_split:-

review_train, review_test, label_train, label_test = train_test_split(meatydf,labeldf, test_size=0.2,random_state=13, shuffle=True)

but I get the ValueError: Found input variables with inconsistent numbers of samples: [4, 21876] which I understand is because of the difference in the shape of X_train_tf and label_train.

enter image description here

I'm not sure how I go about fixing this problem and the internet scouring hasn't helped me very much either, probably because I don't know what I should be looking for being very recently introduced to the topic.

Thanks a ton for your time.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.