I was building a text classifier which takes into account certain features of the text and classifies them into two - "Yes" or "No".
I have trimmed the text, removed stopwords and have applied TFIDF on it and now I'm performing SVM classification on it and getting around 81% accuracy on a dataset of size 25,000 with 80:20 training and test split.
I'd like to include some features in the text and have my classifier predict on that.
Example of how my data might look like:-
The code I was using for predicting the label just using text is:-
labeldf=meatydf['Label] review_train, review_test, label_train, label_test = train_test_split(meatydf['text'],labeldf, test_size=0.2,random_state=13, shuffle=True) X_train_tf = tf_vect.fit_transform(review_train) X_test_tf = tf_vect.transform(review_test) clf = svm.SVC(C=10,gamma=0.001,kernel='linear') clf.fit(X_train_tf,label_train) pred = clf.predict(X_test_tf)
I want to put all the columns of meatydf(text, feature1, feature2 and feature3) as features to help the classifier predict the labels and so I passed the entire dataframe(meatydf) to train_test_split:-
review_train, review_test, label_train, label_test = train_test_split(meatydf,labeldf, test_size=0.2,random_state=13, shuffle=True)
but I get the
ValueError: Found input variables with inconsistent numbers of samples: [4, 21876]
which I understand is because of the difference in the shape of X_train_tf and label_train.
I'm not sure how I go about fixing this problem and the internet scouring hasn't helped me very much either, probably because I don't know what I should be looking for being very recently introduced to the topic.
Thanks a ton for your time.