Timeline for Is a 100% model accuracy on out-of-sample data overfitting?
Current License: CC BY-SA 3.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 9, 2018 at 9:52 | comment | added | Alex | 1)try 50%, 2)jan may be right: your model is genuinely good, 3) try another classifier, mb SVM? | |
| S Feb 9, 2018 at 1:23 | history | edited | Stephen Rauch♦ | CC BY-SA 3.0 |
general language fixes
|
| S Feb 9, 2018 at 1:23 | history | suggested | CommunityBot | CC BY-SA 3.0 |
general language fixes
|
| Feb 9, 2018 at 0:43 | review | Suggested edits | |||
| S Feb 9, 2018 at 1:23 | |||||
| Feb 8, 2018 at 15:29 | comment | added | Milan van Dijck | @Alex I made it about ±30% "ok" and ±70% "bad". The accuracy drops too: 97.10145%, So not much of a difference | |
| Feb 8, 2018 at 15:19 | answer | added | tom | timeline score: 1 | |
| Feb 8, 2018 at 13:41 | history | edited | Stephen Rauch♦ | CC BY-SA 3.0 |
Remove extra commentary
|
| Feb 8, 2018 at 13:16 | comment | added | Alex | That's quite imbalanced. Try using resampling (repeated sampling) to tip the balance in the training set towards OK class (make it 30% for example) and keep the 11/89 ratio in the test/validation sets. What do you get? | |
| Feb 8, 2018 at 10:11 | vote | accept | Milan van Dijck | ||
| Feb 8, 2018 at 9:51 | comment | added | Milan van Dijck |
@Toros91 I have been looking at the predictor importance that the randomforest lays out. for the particular model it is f1: 11.626811 - f2: 14.647147 - f3: 1.797175 - f4: 6.501746
|
|
| Feb 8, 2018 at 9:47 | comment | added | Milan van Dijck | @Alex 11.35% "ok" and 88.65% "bad" | |
| Feb 8, 2018 at 9:43 | comment | added | Alex | Whats the split between classes? | |
| Feb 8, 2018 at 9:39 | comment | added | Toros91 | as "Jan van der Vegt" has suggested make sure that the features are completely correlated with the target variable, will explain with example for better understanding: if you are trying to predict number of years and taking DOB as a feature. Have you done Predictor Importance test? | |
| Feb 8, 2018 at 9:36 | comment | added | Milan van Dijck | @Toros91 added the objective as an edit | |
| Feb 8, 2018 at 9:36 | history | edited | Milan van Dijck | CC BY-SA 3.0 |
additional information
|
| Feb 8, 2018 at 9:28 | comment | added | Toros91 | just take couple of noise records and try testing and see how it performs. how many features do you have? if it is not confidential, can you state your objective? | |
| Feb 8, 2018 at 9:27 | answer | added | Jan van der Vegt | timeline score: 29 | |
| Feb 8, 2018 at 9:26 | comment | added | Milan van Dijck | @Toros91 I haven't tried that specifically but the original "bad" data does contains some very random data. With noise data do you mean randomly generated data? | |
| Feb 8, 2018 at 9:24 | comment | added | Milan van Dijck | @Alex Not exactly but it stays very high like 98,55% | |
| Feb 8, 2018 at 9:19 | comment | added | Alex | Every time you reshuffle, train and test, the accuracy is 100%? | |
| Feb 8, 2018 at 9:15 | comment | added | Toros91 | welcome to the site! Did you try predicting on some noise data? | |
| Feb 8, 2018 at 9:13 | review | First posts | |||
| Feb 8, 2018 at 13:41 | |||||
| Feb 8, 2018 at 9:13 | history | asked | Milan van Dijck | CC BY-SA 3.0 |