While I know it's possible to use StandardScaler on a SparseVector column, I wonder now if this is a valid transformation. My reason is that the output (most likely) will not be sparse. For example, if feature values are strictly positive, then all 0's in your input should transform to some negative value; thus you no longer have a sparse vector. So why is this allowed in Spark, and is it a bad idea to use StandardScaler when you want sparse features?
1 Answer
$\begingroup$
$\endgroup$
The default for the parameter withMean
is False
, so that data won't be centered, just scaled by the standard deviation(s).