# Why does using a standard scalar on my tf idf matrix make it perform better?

I have a TF-IDF matrix transformed on a list of tweets from a data set I am using. I have a pipeline where I initiate a StandardScalar and then next have my SVM with a linear kernel and auto gamma as the classifier algorithm.

Pretty much as done here in the examples section. With the pipeline, the classifier scores an 87 f1 score. Without the pipe, it scores a dismal 53.

Why is this?

I thought TF-IDF values were already two-fold normalised so shouldn't the standard scalar have no effect as it just performs normalisation again?

• stats.stackexchange.com/questions/65047/… This is what i mean by two-fold normalisation. When i use the StandardScalar i have to use it with 'mean = false' because i get this error otherwise "Cannot center sparse matrices: pass with_mean=False. Could you let me know what's going on here, and try help me to give a reason as to why adding the standardscalar increases performance? Jan 11 at 11:09