1
$\begingroup$

I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with and how RF imputation works in PySpark.

$\endgroup$
1
$\begingroup$

You can do the following: use all the other features as input and the missing data as the label.

Train using all the rows that have the column filled with data and classify the others that don't. Use the values predicted by the Random Forest as the value of that field on the subsequent models and transformations.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you @Felipe. How should I train the classifier on rows? Would I have to encode categorical features? $\endgroup$ – moirK Apr 19 '18 at 10:30
  • $\begingroup$ @zimmer I'll point you to the documentation, it's the first place you should go when using a new library. spark.apache.org/docs/latest/api/python/index.html They describe how to build a model and the training set. Regarding categorical data: it depends, but mostly you'll have to use a StringIndexer, here is a simple example: stackoverflow.com/questions/36942233/… $\endgroup$ – Felipe Bormann Apr 19 '18 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.