4
$\begingroup$

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers.

I'd like to use a method that is much faster than it is accurate, because of the size of the data-set and the fact that I don't have much to lose in a speed-accuracy tradeoff.

Running missForest() takes several hours while Hmisc's impute() function gives unsatisfactory results.

What functions in R might be useful in such (or similar) case?


<code>mice_plot</code> output

$\endgroup$
5
  • $\begingroup$ the mice (multiple imputation by chained equations) package is very popular for missing data in R. Its not exactly inexpensive, but it may be a good balance between missForest and impute $\endgroup$
    – TBSRounder
    Jun 13, 2016 at 13:15
  • $\begingroup$ mice has been running for the past 4 hours, appears to be doing better than missForest (efficiency-wise). Are there any metrics to actually compare these imputation methods? $\endgroup$
    – neural-nut
    Jun 13, 2016 at 13:28
  • $\begingroup$ Perhaps this is where I should mention that my computational resources are very limited, hence the emphasis on speed and efficiency. $\endgroup$
    – neural-nut
    Jun 13, 2016 at 13:36
  • $\begingroup$ Probably the best way to compare them would be to make a fake data set with variables that are similar to yours, artificially remove data, run both methods, and see which was closer to the true missing values (rmse/accuracy). Probably not worth it in your limited resources scenario though. $\endgroup$
    – TBSRounder
    Jun 13, 2016 at 14:54
  • 1
    $\begingroup$ You can try avg-filling all columns except one, then predict on that column with the data you do have, and use that model to predict the values that are missing. So you would do this 4 times, with each iteration being predicted on a different column that contains missing values. Not perfect or super-efficient, but might be easy for a simple algorithm like KNN $\endgroup$
    – TBSRounder
    Jun 13, 2016 at 14:57

1 Answer 1

1
$\begingroup$

Take a look at the h20 package https://cran.r-project.org/web/packages/h2o/h2o.pdf.

Everything is designed with parallelization in mind. I've had great success with many of their implementations, in R and Scala.

If you have to do it in R and are going for pure speed I doubt you'll find something faster.

$\endgroup$
1
  • $\begingroup$ I've up-voted because this would have suited my purpose earlier when I was willing to trade-off speed for accuracy, but as it turns out any standard impute function only messes up the classification later on. A regular hmisc::impute works just as well for me in terms of speed. Can I somehow speed-up the mice based imputation? $\endgroup$
    – neural-nut
    Jun 27, 2016 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.