I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 additional records that contain some missing data. Certainly, if a record is missing 90% of all of its feature values, it's best just to remove it - mean imputation would just be adding average records for all features to the training matrix and it can overfit. If a record is only missing 2% of its feature values, then imputation would be great and you can keep it. What do you believe is the correct threshold for deciding between keeping and tossing?

  • $\begingroup$ just to clarify, what does that 300,000 refer to? a single feature that u're going to use in ur model I suppose? $\endgroup$
    – misheekoh
    Aug 5, 2020 at 17:19
  • $\begingroup$ o sorry ill edit it to clarify. I have 300,000 additional records with some missing data. so 900,000 total and 300,000 have missing data and debating which to throw away and which to include. $\endgroup$
    – Evan
    Aug 5, 2020 at 18:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.