Hand-crafted decision tree inspired from learned decision tree

Question

Goal of this question: As I am the only 'machine learning guy' in our group, I wanted to get an outsiders view, that is a sanity check if what I am doing adheres at least to 'decent practices' in machine learning (I know its not best practices :) ).

Problem setup: I am working on a classification task on biomedical signals (detection of hypertension from physiological signals other than blood pressure). Since I do not have too much high quality labeled data for training a powerful classifier (say a larger conv-net), currently my procedure is as follows:

1. Feature engineering (manually engineered features, mainly driven by (physiology) domain insights.

2. Train supervised learning classifiers, in particular tree based algorithms such as random forests and simple decision trees.

Now, since currently I do not have enough high quality labels and I need to ship some classifier soon (in addition as it is a medical application I really care about stability and to some degree interpretability), I thought I could go for a manually built expert system, that is some rule-based system (if-else) using the most relevant features according to the learned random forest or decision tree classifier. For the cutoffs on the features I can use the values learned by the single decision tree. In addition, I would start from a learned decision tree and adapt it (as some learned splits are totally non-sense, i.e., we are in the overfitting regime).

Question: Is this procedure fine as long as I test my (hand-crafted and machine learning inspired) classifier on unseen data? I'd be happy to hear your experiences in similar situations!

Thanks

Answer 1

I did this once, in a position similar to yours.

My constraints were,

• People skeptical of machine learning
• Required interpretability
• Required very fast inference
• Explainable in text documentation
• Implementable in any language

Traditionally, this would have been done by an expert system, so training a decision tree was essentially just one step further.

On my end, all of the work was being done in Python. I tried a handful of ML models, with and without hand-crafted features. A single decision tree performed nearly as well as anything I tried, and outperformed a fully handcrafted expert system. I then created an "inference model" in the form of if-statements for each split in the tree to confirm performance was equal to the decision tree itself. I even manually ignored some low entropy splits or where a single feature was underutilized, finding that for the same number of branches, a larger trained tree manually retaining fewer nodes outperformed a small tree where I replicated it exactly.

Some of the splits were unintuitive, but that's partially the point of throwing a model at it. Be sure to really dig into the failure modes of every alternative to help everyone involved have confidence in your work.

Given the constraints, it was a great solution to the problem.