Timeline for Multiple models have extreme differences during evaluation
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 8, 2021 at 15:06 | vote | accept | Egor | ||
| Oct 8, 2021 at 14:14 | comment | added | Egor | I spent last night testing exactly that, and that was the problem. I had duplicates in both sets. Ugh, I knew it was too good to be true haha. Thank you in any case; that was a good lesson for me | |
| Oct 8, 2021 at 12:32 | comment | added | Erwan | ... This can cause a bias in the evaluation because the test set contains instances which have been seen during training (this would be a case of data leakage). | |
| Oct 8, 2021 at 12:31 | comment | added | Erwan | @EgorIsakson in general it's best to have a large number of instances with a moderate number of features in order to avoid overfitting. With 100k instances and only 6 features you're really on the safe side, if you have other features available you could easily add them. However as usual this depends on the data, that's why I mentioned the point about possible duplicates: the data size might seem high but actually be equivalent to a small dataset if it's not "diverse enough". By themselves the duplicates are not a problem, except if the data is split randomly between training and test set: ... | |
| Oct 8, 2021 at 3:59 | comment | added | Egor | Erwan, I highly appreciate your thoughtful answer!! Edited my question with the accurate data. My random forest doesn't seem to overfit. I ran a 10k fold validation, and all 10 accuracies, f1's, etc were consistently high. You are right about K-nn; among least performing models, this one was doing the best with a few thousand positives guessed. As for the dataset size, you might be right about the duplicates, but if they are there for both classes could they cancel out? Also, could you share your thoughts on max feature amount for 100k instances? All the features are numerical. | |
| Oct 8, 2021 at 2:12 | history | answered | Erwan | CC BY-SA 4.0 |