4
$\begingroup$

I have what seems to me to be a slightly complex version of a decision tree problem, that can't figure out how to model, and I'm trying to avoid the "just dump it into an NN" solution.

I have a bunch of expert opinions (from either people or algorithms) where the features dimensions overlap. Both features (inputs) and outputs (decision) are categorical, but I also have a certainty in the decision (let's say 0-1).

For example, let's say that the input features are a,b,c,d,e and there is one output (decisions) from among v,w,x,y,z:

Features   Decision  certainty
 a,b,c       w          0.5
 b,c,d,e     z          0.4
 a,c,e,f     v          0.75
 ...

Without the certainty, this would obviously just be a trivial categorical decision tree problem. However, the certainty both simplifies and complexifies the problem. It simplifies it bcs if the features->decision data is inconsistent (which it is!), you'd have no output at all, so the certainty saves me from that failure mode, but on the otherhand, I'm not sure what to correctly do with the certainty.

The certainties suggest Bayesian combination, and if the data was unidimensional this would be trivial as well. So I'm kind of caught half way between a decision tree and a bayesian model. An obvious cop out is just to dump it into an NN (or even just an NB or regression), using the certainties either as outputs on the categories, or doing something dumb like replicating the I/O pairings per the (un)certainty.

Thanks in advance for any suggestions.

$\endgroup$
1
$\begingroup$

However, the certainty both simplifies and complexifies the problem. It simplifies it bcs if the features->decision data is inconsistent (which it is!), you'd have no output at all, so the certainty saves me from that failure mode, but on the otherhand, I'm not sure what to correctly do with the certainty.

I do not understand what you wanted to say with this.

Anyway, the most straightforward way to consider the certainty is to use it as a weight (or exposure) and use the methods that support it, e.g. GLMs or GBMs.

For example, the concept of weights is explained in this blog post

| improve this answer | |
$\endgroup$
1
$\begingroup$

Just so you know, in this case, even the neural network can't help you. For NNs to work, you need to have lots of learnable patterns in the data (meaning data without uncertainity/ irreducible error). If the data itself has a lot of uncertainty in the outcome( i.e. probabilistic outcomes) itself, its very hard of NNs to optimize that. (For more on this, consider modeling following problems with NNs: XOR(with probability =1) vs XOR(with probabilities <1)).

In this case, your best bet might be to use regression. Let us know how it folds out, thanks.

| improve this answer | |
$\endgroup$
0
$\begingroup$

It depends a bit on what uncertainty measures here.

If it means that there is a probability p that it is the given outcome, then it's not that difficult to approach. If you have 4 outcomes and you know that outcome a has 60%, then the other three together have 40%. You could either spread this 40% uniformly over the other three categories, use the prior probabilities to spread it out or use another model to predict how it would be spread out. Once you have a model over the other probabilities, you could use a neural network classification approach where you pass these probabilities instead of one hot encoded targets. Directly passing them to a tree based method will not work although XGBoost multiclas classification might be able to be tweaked to handle this. That said, another approach would be to then resample your dataset to make it significantly bigger, and sample the target from this distribution.

If uncertainty is some metric that is more a feeling than a direct mathematical thing, weighting your loss is probably better but it might not necessarily be best to take this linearly.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.