I'm working on a Sentiment Analysis task using TF-IDF to build my features and SVC as the classifier.

My goal is to make my model to classify the sentiment of all my dataset. I already designed my model with training and test data, splitting my dataset into training and test sets, but now I want my model to give the labels for each element of the dataset.

For that, I made the following steps

  1. Applied the TF-IDF fit_transform() to my train dataset
  2. Applied the TF-IDF transform() to my entire dataset
  3. Trained a SVC with my train dataset
  4. Predicted the sentiment of my entire dataset
  5. Got the classifications and the probabilities for each class
from sklearn.svm import SVC
from sklearn import metrics

Tfidf_X_train = Tfidf.fit_transform(X_train['text']).toarray()

Tfidf_df  = Tfidf.transform(df_preproc['text']).toarray()

svc_classifier = SVC(probability=True)
svc_classifier.fit(Tfidf_X_train, y_train)
y_pred = svc_classifier.predict(Tfidf_df)
class_probabilities = svc_classifier.predict_proba(Tfidf_df)

The problem is when I look what class_probabilities or y_pred returns it gives me duplicated lines, as if it classified the same text more than once, even I knowing that each text is unique in the dataset. Like the following example:

enter image description here

Where am I doing wrong?

  • 2
    $\begingroup$ Are you sure it is duplication? It could also be the cases that after TfIdf transformation, 2 distinct rows have exactly the same representation because they contained mostly similar words. $\endgroup$ – hssay Aug 24 '20 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.