Standard Scaler drops accuracy significantly in Scala Spark

Question

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly.

Without the accuracy is ~90% (on training set), 77% (on testing set)

With Normalizer or Standard Scaler is 19% (on training), 0% (on testing)

I have the feeling that it's not related to the normalization per se, but more like a code bug.

Here is part of the code:

val labelIndexer = new StringIndexer()
  .setInputCol("label")
  .setOutputCol("indexedLabel")
  .fit(assemblerDF)

val featureIndexer = new VectorIndexer()
  .setInputCol("features")
  .setOutputCol("indexedFeatures")
  .setMaxCategories(4) //if column has more than 4 categories, then use it as continious instead
  .fit(assemblerDF)


val labelConverter = new IndexToString()
  .setInputCol("prediction")
  .setOutputCol("predictedLabel")
  .setLabels(labelIndexer.labels)

val scaler = new StandardScaler()
  .setInputCol("indexedFeatures")
  .setOutputCol("scaledFeatures")
  .setWithStd(true)
  .setWithMean(false)

val ml = new LogisticRegression()
  .setMaxIter(15)
  .setFeaturesCol("scaledFeatures")
  .setLabelCol("indexedLabel")

val pipeline = new Pipeline().setStages(Array(labelIndexer, featureIndexer, scaler, ml, labelConverter))

Where then I pass the Pipeline into a CrossValidator. Any idea what is going wrong here?

Answer 1

As @n1k31t4 said in the comments, it's the .setWithMean(false) call (should be true).

Moreover, you don't really need the StandardScaler at all, since LogisticRegression will perform standardization by default (see the standardization param).