Computational vs intuitionistic or expert-based information gain in decision trees?

Computational vs intuitionistic or expert-based information gain in decision trees?

This confuses me.

Plenty of literature on how information gain can be used when it's calculated computationally. But what if there's a competing sense of "intuitionistic (or expert-based) information importance"? That is, the researcher has an intution about relative importances and this may not actually be conveyed in the training set. Or some of it may be lost by the model.

If one'd use computational methods to infer good split points, then it's possible that these match the training set, but not necessarily the intuition.

It's also possible that the intuitional approach would later prove inaccurate in some sense, if new observations would come that display computational information gains that suggest readjusting intuitionistic bounds.

So is there some middle-ground to combine these two views?

• Which two views? Do you have any actual reference to this "intuitionistic" approach you try to describe here? Because it would seem that you simply wonder if it exists, hence asking about some middle ground between an existing established approach and a speculative one that you are not even sure it exists does not make much sense, does it? I would perfectly understand asking for confirmation if it indeed exists, but a "middle ground"? Commented Jul 11, 2021 at 21:19
• @desertnaut You can also call it "expert opinion" or "tacit knowledge" or the sorts. Basically it refers to the informed opinion that an expert on the topic has about the information of each variable, based on his/her experience. This opinion is not just some numerical idea about whether variables are related, but based on e.g. what literature the person has read, what kinds of real-world situations he/she has seen. Commented Jul 12, 2021 at 5:49

The core principle in supervised machine learning is that the training data is a representative sample of the true distribution (i.e. the possibly infinite full set of instances that could happen).

Under this assumption, the intuition and the numerical information gain (or other statistical measure) are expected to be more or less in agreement, because if they disagree it can only mean that:

• Either the intuition was wrong, because if a feature is intuitively important then there should be evidence of that in the data.
• Or the data is not a proper representative subset of the true distribution (insufficient or noisy training data).

But it's important to keep in mind that a dataset is never perfect and that an intuition is, well, just an intuition. So for example it would be common that the top 3 features A,B,C according to intuition are not exactly the top 3 features according to IG, it might be something reasonably close like B,D,E,A,C for instance.

If the data doesn't match the intuition at all, it's worth investigating why. However in general it would be a bad idea to overrule the computed IG value and force the use of "intuitively strong" features, because this is clearly not optimal according to the data.

• Yeah well that's why I called it competing. If the model contradicts the intuition, but the researcher trusts the intution. Then this confuses as to what to look for. An error in the model or an error in the intuition? Commented Jul 11, 2021 at 19:39
• Some sources claim that computational methods are more useful for large data sets in general. Because for small, it's often possible to make strong human interpretations about the data, because one can essentially read each observation. But if one's to study 100000 observations, then managing information of all of them is not feasible, manually. But then one could expect that strong observations on smaller samples "should perhaps hold" on the larger as well. Yet how is one supposed to infer, whether the intuitions hold or the model is giving more correct info to correct intuitions? Commented Jul 11, 2021 at 19:44
• There's for example this medical example, where it's explained that "expert opinion" is somewhat equivalently significant, since it references different sorts of information: the expert's experience and the literature that he/she knows. The limitations of decision trees and automatic learning in real world medical decision making pubmed.ncbi.nlm.nih.gov/9555627 . Trusting the model in such case is not reliable. But nor is necessary trusting the expert only either. Just using a decision tree there will give causality that's not professionally supported. Commented Jul 11, 2021 at 19:48
• @mavavilj first point: one shouldn't be looking for any particular type of error but rather trying to the most likely explanation based on evidence. This might require setting up a controlled experiment with proper evaluation in order to compare the two methods objectively, for example. The question of "which feature" is not essential in a prediction problem, it's indeed very likely that a human expert uses a different kind of evidence and a different kind of reasoning compared to a ML algorithm. Whether their final prediction/conclusion is correct is how to compare the two methods. Commented Jul 11, 2021 at 22:04
• In general a ML prediction is not a "decision", it's a prediction based on some calculations using the training set as a reference. The performance of a model should be evaluated as reliably as possible, that's how one knows how much to trust its predictions. Commented Jul 11, 2021 at 22:08