4
$\begingroup$

Unfortunately trying to google or research null values in machine learning always brings up pages trying to teach you how to impute the values instead, but I'm trying to find models that can handle null values as input. The only one I've found currently is XGBoost, which is a gradient descent algorithm. I believe tree based algorithms should in theory handle null values as well, but I'm not sure if that generalizes to all tree based algorithms or if some work better than others.

Some background, I have created multiple datasets, including ones where the NaNs ARE replaced. Those can be used to train models that can't handle null values, like neural networks.

Due to the sparseness and nature of the data, imputing with mean or median is going to introduce a LOT of bias and doesn't make sense. Just want to get that out there since that's the first thing everyone suggests. Also, the missing data actually represents a situation where a patient didn't get a specific lab test done, which is useful information of its own.

So back to the original question, besides XGBoost, are there other models that handle null values as inputs? Does that generalize to ALL gradient descent algorithms or tree algorithms or just specific ones? Is there a preferred model for a situation where you want to keep null values?

$\endgroup$
2
  • $\begingroup$ Do you know how XGBoost handle missing values? In my knowledge, it handles it internally, meaning it imputes them with a default method, like Catboost see the last FAQ catboost.ai/docs/concepts/faq.html. I do not this is on the model per se, it s more on the implementation of it. Well there are ideas like researchgate.net/post/…, how to estimate missing values like we do in Entity Encoding (Embedding). $\endgroup$ – TwinPenguins Jan 29 '20 at 6:03
  • $\begingroup$ I would also like to add that Random Forest accepts null values and can have a rather robust performance. $\endgroup$ – Georgiy May 11 '20 at 18:42
2
$\begingroup$

LightGBM and XGBoost Libraries can handle missing values

  • LightGBM: will ignore missing values during a split, then allocate them to whichever side reduces the loss the most
  • XGBoost: the instance is classified into a default direction (the optimal default directions are learnt from the data somehow)

Finally, it is NOT a general property of gradient descent algorithms or tree algorithms. Only specific implementations of these algorithms have this property.

Personally, I believe that LightGBM and XGBoost can handle missing values effectively, in case one wants to keep them

$\endgroup$
2
$\begingroup$

LightGBM by default handles null values by setting them to zero. You can also have it assume zeros are null values by setting zero_as_missing=true. So, while it handles them in the back end, it doesn't do anything different to you imputing with zeros.

here is the documentation : https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html

Personally, I prefer to have control over how my nulls are handled so I don't generally use this feature.

It looks like you can also have it ignore missing values, but you have to set this expressly as an option: use_missing=false. I haven't tried this.

One of the things I often do when using tree based algorithms is set nulls to a value that is not possible within the non-null dataset, e.g. -1 for things that should always be positive. This acts as a null flag rather than treating the null as a zero. You have to be careful how you do this.

I would test a range of options and see which one produces the most predictive model - you may find that imputing with means or medians is the best thing for your dataset.

$\endgroup$
1
  • $\begingroup$ I don't think that first sentence is true. Can you quote where in the documentation that's described? $\endgroup$ – Ben Reiniger Sep 3 '20 at 14:24
0
$\begingroup$

Since you are asking which packages, H2o GBM, Rpart and R gbm handle missing as well. Through different ways such as surrogate variables, another category, 3-way split (left, right, missing).

You should also ask what if there is missing in a feature during scoring that had no missing during training - how will that be handled? Just handling missing during training may not cover all of the cases.

Anyway you go, you might want to use an indicator variable as well. Sometimes the signal is in something vs nothing, rather than lots of values and nothing. There is a thought that trees are smart enough to find the signal if it is missing vs everything else by using a flag in the data, but I have seen the opposite so many times. Especially as you mentioned the data is sparse and there may be signal in the patient getting a test or not. So make that explicit and give the algorithm a better chance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.