I have a collection of 10 features (all numerical) and a single binary outcome variable. I need to train a binary classification model, find the best features and compute thresholds for each feature.

To find the best subset of features is easy, I use logistic regression with L1 penalisation and it works nicely. However, the next step is to find thresholds for each subselected feature: if the values of a feature i, j,...k are above/below than some numerical values, than chances to be in the class A (rather than in B) are higher.


Consider, we found that out of 10 original features, there are L1 leaves only 3: F1, F4,F7. And if you take three unseen (new) data with some values {F1_i, F4_i, F7_i}, where

F1 > 1.23
F4 < 7.15
F7 > 2.74

Then the new data point {F1_i, F4_i, F7_i} belongs to class A. Here {1.23, 7.15, 2.74} are thresholds.

I tried to explain the problem as clear as possible, but if it isn't, please let me know.

QUESTION What will be the best approach to solve this problem? How to compute thresholds?


1 Answer 1


You are describing every binary classifier. However, you are missing a key point. If your classes are separable by the value of just ONE feature, you can do what you're saying and find e.g. F1 > 1.23 as a threshold. If classification involves a combination of features, you will need to describe some combination of thresholds for each feature, or (equivalently) some relationship between the features that tells you about the class label. It's the job of every binary classifier to do exactly this - they just do it in different ways. See for example this post. Your desire to have a combination of fixed sets of thresholds will only work if you can have a set of thresholds that will encompass/describe/classify every combination of feature values.

If you want a set of easy-to-read thresholds like you mention, you should read about decision tree classifiers. They'll do something like what you want - but will also ensure that you provide a class label for every possible combination fo features values. The nice about about decision trees is that they'll let you leave out your current feature selection step - they just do it for you by (1) picking the feature that best discriminates the two classes overall, (2) picking the threshold value of that feature that gives the most information about the class label (usually), and (3) repeating (1-2) several times.

  • $\begingroup$ many thanks for your answer! I'm working with medical data and the idea is to develop a simple classifier which can distinguish presence and absence of a disease. More importantly, it should be accessible by clinicians, which have no idea about classifiers or even logistic regression. The problem I'm solving (btw, I'm not sure whether it is solvable) is: to find a minimal set of features (blood test components) and provide thresholds for each. This approach will allow clinicians simply check whether the blood results are above/below these thresholds and estimate probabilities $\endgroup$ Feb 6, 2018 at 8:48
  • $\begingroup$ of having or not having a disease. I think you suggestion to look at Trees is exactly what I need, as trees (a single shallow tree, not Random Forest) provide rectangular decision boundaries (independent thresholds) and it may solve my problem. If you have some links/suggestions, please let me know. Many thanks again! $\endgroup$ Feb 6, 2018 at 8:51
  • 1
    $\begingroup$ Hope it was helpful! Decision trees are frequently used in the medical setting exactly because they are easily interpretable. There is an R implementation of decision tree algorithms called 'rpart' - you can read a description of the algorithm and its use on medical data here: cran.r-project.org/web/packages/rpart/vignettes/longintro.pdf $\endgroup$
    – tom
    Feb 6, 2018 at 15:38
  • $\begingroup$ brilliant! I'm going to explore the packages. $\endgroup$ Feb 6, 2018 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.