Is it possible to do hard-coded decision tree on some variables and random forest / something on the remaining ones?

The situation seems that for some variables it's possible to draw strong empirical assumptions, but for others their "relative importance" seems more random.

So e.g.

Researcher is certain that splitting X1 > 5 and X2 < 3 gives best information, since they are empirically sound splits e.g. based on stakeholder views. And X1, X2 are more important than X3, X4, X5, since X3, X4, X5 are redundant, if X1 or X2 don't exist.

Thus the model could essentially be based on X1, X2 only , but X3, X4, X5 should add explanatory power. Yet their relative importances are not known. Using the decision tree to them might be prone to model inaccuracies due to random forest or something perhaps offering better reduction in overfitting etc.

  • $\begingroup$ Definetely, yes, the questions is how to do in pythonic way. Or whatever you are coding $\endgroup$ Jul 12 at 7:52
  • $\begingroup$ @CarlosMougan I don't find it definite, because how to e.g. balance the metrics of the model, if part of it is decision tree and part is RF. Then what's the total accuracy for example? $\endgroup$
    – mavavilj
    Jul 12 at 8:25
  • $\begingroup$ @mavavilj performance (accuracy or other) can be calculated for any predicting system, including a human making predictions for example. It is not related to how the predicting system is made. As I said in your other question you probably have some confusion about the distinction between a predicting system and evaluation. $\endgroup$
    – Erwan
    Jul 13 at 20:13
  • $\begingroup$ @Erwan I understand the technical difference, but I'm confused about the difference of accuracy gained by "expert opinion" and "computational methods", as in this example: pubmed.ncbi.nlm.nih.gov/9555627. So, if we have "strong expert opinion" on X1, X2 above, but not on X3, X4, X5. Then this could motivate one to use different methods on them. While the global prediction should still be aggregated from all. Using e.g. RF for all could distort expert opinion on X1, X2 in favor of "numerical idea of truth". Evaluation is also pointless, if the model is faulty(?) $\endgroup$
    – mavavilj
    Jul 14 at 4:56
  • $\begingroup$ @mavavilj this abstract doesn't give any detail about what the authors do. More importantly, selecting the features is like choosing a method to solve the problem, as you said. But evaluation is not about the method, it's about determining how well the method works. Evaluation can be applied to any system, whether it's good or bad, including a system which gives random predictions or always the same output. If the system is faulty, then evaluation should return a low performance score, that's it. So it must be possible to compare any system by their performance. $\endgroup$
    – Erwan
    Jul 14 at 9:45

The important point here is the distinction between rule-based and data-driven:

  • A rule-based predicting system is an algorithm which calculates the target variable based on rules which have been predetermined and implemented by a human expert.
  • A data-driven predicting system is an algorithm which is first trained on some labelled training data in order to automatically determine the "rules".

Both types of systems can be used to predict the target variable for some new instance. Both types of systems can (and should) be evaluated using some labeled test set.

So in general yes, one can hard-code a decision tree. If it's entirely hard-coded then it's a rule-based system.

In theory one can use a hybrid method to build a tree using some human-determined rules and the rest based on data. However in this case there's no established algorithm which says how the two types of "rules" should be combined (to the best of my knowledge). For example one could create the top of the tree manually and then let the learning algorithm determine the rest of the tree. Or one could create multiple trees, some rule-based and some data-driven, then use an ensemble method to combine their predictions.

But it's important to realize that this kind hybrid method could lead to inconsistencies: if the expert decides to give priority to a rule which is not supported (or doesn't have a high importance) in the data, then it's likely that the resulting model will perform poorly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.