0
$\begingroup$

I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 additional records that contain some missing data. Certainly, if a record is missing 90% of all of its feature values, it's best just to remove it - mean imputation would just be adding average records for all features to the training matrix and it can overfit. If a record is only missing 2% of its feature values, then imputation would be great and you can keep it. What do you believe is the correct threshold for deciding between keeping and tossing?

Improve this question
$\endgroup$
2
  • $\begingroup$ just to clarify, what does that 300,000 refer to? a single feature that u're going to use in ur model I suppose? $\endgroup$
    – misheekoh
    Aug 5, 2020 at 17:19
  • $\begingroup$ o sorry ill edit it to clarify. I have 300,000 additional records with some missing data. so 900,000 total and 300,000 have missing data and debating which to throw away and which to include. $\endgroup$
    – Evan
    Aug 5, 2020 at 18:13

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.