1
$\begingroup$

I was building a text classifier which takes into account certain features of the text and classifies them into two - "Yes" or "No".

I have trimmed the text, removed stopwords and have applied TFIDF on it and now I'm performing SVM classification on it and getting around 81% accuracy on a dataset of size 25,000 with 80:20 training and test split.

I'd like to include some features in the text and have my classifier predict on that.

Example of how my data might look like:-

enter image description here

The code I was using for predicting the label just using text is:-

labeldf=meatydf['Label]


review_train, review_test, label_train, label_test = train_test_split(meatydf['text'],labeldf, test_size=0.2,random_state=13, shuffle=True)

X_train_tf = tf_vect.fit_transform(review_train)
X_test_tf = tf_vect.transform(review_test)

clf = svm.SVC(C=10,gamma=0.001,kernel='linear')
clf.fit(X_train_tf,label_train)
pred = clf.predict(X_test_tf)

I want to put all the columns of meatydf(text, feature1, feature2 and feature3) as features to help the classifier predict the labels and so I passed the entire dataframe(meatydf) to train_test_split:-

review_train, review_test, label_train, label_test = train_test_split(meatydf,labeldf, test_size=0.2,random_state=13, shuffle=True)

but I get the ValueError: Found input variables with inconsistent numbers of samples: [4, 21876] which I understand is because of the difference in the shape of X_train_tf and label_train.

enter image description here

I'm not sure how I go about fixing this problem and the internet scouring hasn't helped me very much either, probably because I don't know what I should be looking for being very recently introduced to the topic.

Thanks a ton for your time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.