1
$\begingroup$

I have a TF-IDF matrix transformed on a list of tweets from a data set I am using. I have a pipeline where I initiate a StandardScalar and then next have my SVM with a linear kernel and auto gamma as the classifier algorithm.

Pretty much as done here in the examples section. With the pipeline, the classifier scores an 87 f1 score. Without the pipe, it scores a dismal 53.

Why is this?

I thought TF-IDF values were already two-fold normalised so shouldn't the standard scalar have no effect as it just performs normalisation again?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I am not sure what you mean by two-fold normalized. TFIDF values are not normalized per se. They will fall within a somewhat constant range of values, but this completely depends on the dataset.

The StandardScaler performs normalization by removing the mean and scaling to unit variance. Since TFIDF values may vary from dataset to dataset, using the StandardScaler will have an effect on the values, which explains the difference in performance.

Improve this answer
$\endgroup$
2
  • $\begingroup$ stats.stackexchange.com/questions/65047/… This is what i mean by two-fold normalisation. When i use the StandardScalar i have to use it with 'mean = false' because i get this error otherwise "Cannot center sparse matrices: pass with_mean=False. Could you let me know what's going on here, and try help me to give a reason as to why adding the standardscalar increases performance? $\endgroup$
    – Synikk
    Jan 11, 2021 at 11:09
  • $\begingroup$ The TFIDF output is a sparse matrix, you need to make it dense for StandardScaler to work evidently (docs.scipy.org/doc/scipy/reference/generated/…). As to why it performs better. It depends on the dataset. It isn't always true, though scaling generally tends to provide better results $\endgroup$
    – Valentin Calomme
    Jan 11, 2021 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.