I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly.

Without the accuracy is ~90% (on training set), 77% (on testing set)

With Normalizer or Standard Scaler is 19% (on training), 0% (on testing)

I have the feeling that it's not related to the normalization per se, but more like a code bug.

Here is part of the code:

val labelIndexer = new StringIndexer()

val featureIndexer = new VectorIndexer()
  .setMaxCategories(4) //if column has more than 4 categories, then use it as continious instead

val labelConverter = new IndexToString()

val scaler = new StandardScaler()

val ml = new LogisticRegression()

val pipeline = new Pipeline().setStages(Array(labelIndexer, featureIndexer, scaler, ml, labelConverter))

Where then I pass the Pipeline into a CrossValidator. Any idea what is going wrong here?

  • $\begingroup$ well if your data is not huge, experiment with python there debug is quite easy, and you can also assess your model easily. $\endgroup$ – JAbr Jan 13 '19 at 10:22
  • $\begingroup$ how many target classes do you have? Could 19% be the accuracy of random guess? $\endgroup$ – n1k31t4 Jan 13 '19 at 16:16
  • $\begingroup$ It’s a two class classification model. $\endgroup$ – Tasos Jan 13 '19 at 16:16
  • 1
    $\begingroup$ I'm not a spark/scala person, but why do you give .setWithMean(false)? For normal scaling, you should scale both meand and variance. Also, getting 0% accuracy on testing in one of your attempts sounds like a bug, given a binary classification. Additionally, it seems you perform the labelConverter on a different column to your actual predictions (indexedLabel) - or is some code missing? $\endgroup$ – n1k31t4 Jan 13 '19 at 16:19
  • $\begingroup$ What are your features like? Are some of them high-cardinality categorical variables? Are any of them interval variables in which the frequency of appearance doesn't follow the numerical order? $\endgroup$ – Thomas Cleberg Jan 15 '19 at 22:07

As @n1k31t4 said in the comments, it's the .setWithMean(false) call (should be true).

Moreover, you don't really need the StandardScaler at all, since LogisticRegression will perform standardization by default (see the standardization param).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.