1
$\begingroup$

I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 entries as they contained significant missing data. When I redo the analysis except use mean, median, and mode imputation methods to fill in the missing data entries and repeat the training/testing - my accuracy drops by 4%.

Why is this? I thought more data would be better or at the least stay the same. Does this imply the imputation was inaccurate? Or perhaps I was biasing the sampling process by only selecting records with no missing entries the original time? How could I possibly know which case since I have no way of knowing how accurate the imputed values are with the true values that were missing?

Improve this question
$\endgroup$
2
  • $\begingroup$ Have you tried training your model with different data sizes only using the 300000 good rows to select from? Maybe try 50000, then 100000, then 200000 and see how much the model improves as the dataset grows. Also, you can try different imputation methods such as KNNImputer, IterativeImputer, or fancyimpute/MICE. As Brian mentions, it appears the missing data is not randomly missing, which makes this dataset more complicated to work with. Also, in general, more data is not always better, it depends on the situation. Reference: scikit-learn.org/stable/modules/impute.html $\endgroup$
    – Donald S
    Aug 5, 2020 at 0:48
  • $\begingroup$ More data is only good when the data is of good quality, thus give us more information. If instead the data is of low quality, we may end up having more noise, thus the performance will be dragged down instead. $\endgroup$
    – Victor Luu
    Aug 5, 2020 at 5:31

1 Answer 1

Reset to default
1
$\begingroup$

That is a clue your data is missing-not-at-random.

The missing data is a result of a systemic process that also effects predicting target performance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I suspected that, and the fact that there is a higher percentage of one of the classes in the missing data group than in the complete data group. But how do you know that the model isn't causing imputed values to overfit to the new data and thus decreasing accuracy by the imputation rather than by the missing-not-at-random problem? $\endgroup$
    – Evan
    Aug 4, 2020 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.