Timeline for Standard Scaler drops accuracy significantly in Scala Spark
Current License: CC BY-SA 4.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 20, 2019 at 14:45 | vote | accept | Tasos | ||
| Feb 20, 2019 at 14:44 | answer | added | Manu Valdés | timeline score: 0 | |
| S Jan 21, 2019 at 10:01 | history | bounty ended | CommunityBot | ||
| S Jan 21, 2019 at 10:01 | history | notice removed | CommunityBot | ||
| Jan 17, 2019 at 18:54 | comment | added | Brian Spiering | Have you tried manually inspecting the DataFrames? A visual inspect might show should odd about the scaled features. | |
| Jan 15, 2019 at 22:07 | comment | added | Thomas Cleberg | What are your features like? Are some of them high-cardinality categorical variables? Are any of them interval variables in which the frequency of appearance doesn't follow the numerical order? | |
| Jan 13, 2019 at 16:19 | comment | added | n1k31t4 |
I'm not a spark/scala person, but why do you give .setWithMean(false)? For normal scaling, you should scale both meand and variance. Also, getting 0% accuracy on testing in one of your attempts sounds like a bug, given a binary classification. Additionally, it seems you perform the labelConverter on a different column to your actual predictions (indexedLabel) - or is some code missing?
|
|
| Jan 13, 2019 at 16:16 | comment | added | Tasos | It’s a two class classification model. | |
| Jan 13, 2019 at 16:16 | comment | added | n1k31t4 | how many target classes do you have? Could 19% be the accuracy of random guess? | |
| Jan 13, 2019 at 10:22 | comment | added | JAbr | well if your data is not huge, experiment with python there debug is quite easy, and you can also assess your model easily. | |
| S Jan 13, 2019 at 8:51 | history | bounty started | Tasos | ||
| S Jan 13, 2019 at 8:51 | history | notice added | Tasos | Draw attention | |
| Jan 12, 2019 at 8:39 | history | edited | Tasos | CC BY-SA 4.0 |
deleted 3 characters in body
|
| Jan 11, 2019 at 8:36 | history | asked | Tasos | CC BY-SA 4.0 |