I have a dataset which contains both text and numeric features.
I have encoded the text ones using the TfidfVectorizer from sklearn.
I would now like to apply logistic regression to the resulting dataframe.
My issue is that the numeric features aren't on the same scale as the ones resulting from tfidf.
I'm unsure about whether to:
scale the whole dataframe with StandardScaler prior to passing to a classifier;
only scale the numeric features, and leave the ones resulting from tfidf as they are.
The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is.
According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange:
(it's) (...) usually is a two-fold normalization.
First, each document is normalized to length 1, so there is no bias for longer or shorter documents. This equals taking the relative frequencies instead of the absolute term counts. This is "TF".
Second, IDF then is a cross-document normalization, that puts less weight on common terms, and more weight on rare terms, by normalizing (weighting) each word with the inverse in-corpus frequency.
Tf-Idf is meant to be used in its raw form in an algorithm. Other numerical values are the ones that could be normalized if the algorithm needs normalization or the data is just too small. Other options can be using algorithms resistant to different ranges and distributions like tree based models or simply using regularization, it's up to the cross-validation results really.
But categorical features like bag-of-words, tf-idf or other nlp transformations should be left alone for better results.
However, there is also the idea of normalizing one-hot coded variables as something that can be done as a standard step same as in other datasets. And it's presented by a prominent figure in the field of statistics.
