Goal of this question: As I am the only 'machine learning guy' in our group, I wanted to get an outsiders view, that is a sanity check if what I am doing adheres at least to 'decent practices' in machine learning (I know its not best practices :) ).
Problem setup: I am working on a classification task on biomedical signals (detection of hypertension from physiological signals other than blood pressure). Since I do not have too much high quality labeled data for training a powerful classifier (say a larger conv-net), currently my procedure is as follows:
Feature engineering (manually engineered features, mainly driven by (physiology) domain insights.
Train supervised learning classifiers, in particular tree based algorithms such as random forests and simple decision trees.
Now, since currently I do not have enough high quality labels and I need to ship some classifier soon (in addition as it is a medical application I really care about stability and to some degree interpretability), I thought I could go for a manually built expert system, that is some rule-based system (if-else) using the most relevant features according to the learned random forest or decision tree classifier. For the cutoffs on the features I can use the values learned by the single decision tree. In addition, I would start from a learned decision tree and adapt it (as some learned splits are totally non-sense, i.e., we are in the overfitting regime).
Question: Is this procedure fine as long as I test my (hand-crafted and machine learning inspired) classifier on unseen data? I'd be happy to hear your experiences in similar situations!
Thanks