I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly.
Without the accuracy is ~90% (on training set), 77% (on testing set)
With Normalizer or Standard Scaler is 19% (on training), 0% (on testing)
I have the feeling that it's not related to the normalization per se, but more like a code bug.
Here is part of the code:
val labelIndexer = new StringIndexer() .setInputCol("label") .setOutputCol("indexedLabel") .fit(assemblerDF) val featureIndexer = new VectorIndexer() .setInputCol("features") .setOutputCol("indexedFeatures") .setMaxCategories(4) //if column has more than 4 categories, then use it as continious instead .fit(assemblerDF) val labelConverter = new IndexToString() .setInputCol("prediction") .setOutputCol("predictedLabel") .setLabels(labelIndexer.labels) val scaler = new StandardScaler() .setInputCol("indexedFeatures") .setOutputCol("scaledFeatures") .setWithStd(true) .setWithMean(false) val ml = new LogisticRegression() .setMaxIter(15) .setFeaturesCol("scaledFeatures") .setLabelCol("indexedLabel") val pipeline = new Pipeline().setStages(Array(labelIndexer, featureIndexer, scaler, ml, labelConverter))
Where then I pass the Pipeline into a CrossValidator. Any idea what is going wrong here?