I'm working on a Sentiment Analysis task using TF-IDF to build my features and SVC as the classifier.
My goal is to make my model to classify the sentiment of all my dataset. I already designed my model with training and test data, splitting my dataset into training and test sets, but now I want my model to give the labels for each element of the dataset.
For that, I made the following steps
- Applied the TF-IDF fit_transform() to my train dataset
- Applied the TF-IDF transform() to my entire dataset
- Trained a SVC with my train dataset
- Predicted the sentiment of my entire dataset
- Got the classifications and the probabilities for each class
from sklearn.svm import SVC from sklearn import metrics Tfidf_X_train = Tfidf.fit_transform(X_train['text']).toarray() Tfidf_df = Tfidf.transform(df_preproc['text']).toarray() svc_classifier = SVC(probability=True) svc_classifier.fit(Tfidf_X_train, y_train) y_pred = svc_classifier.predict(Tfidf_df) class_probabilities = svc_classifier.predict_proba(Tfidf_df)
The problem is when I look what class_probabilities or y_pred returns it gives me duplicated lines, as if it classified the same text more than once, even I knowing that each text is unique in the dataset. Like the following example:
Where am I doing wrong?