1
$\begingroup$

I understand how a decision tree is constructed (in the ID3 algorithm) using criterion such as entropy, gini index, and variance reduction. But the formulae for these criteria do not care about optimization metrics such as accuracy, recall, AUC, kappa, f1-score, and others.

R and Python packages allow me to optimize for such metrics when I construct a decision tree. What do they do differently for each of these metrics? Where does the change happen?

Is there a pattern to how these changes are done for different classification/regression algorithms?

$\endgroup$
3
  • $\begingroup$ Could you provide an example of packages optimizing such metrics? $\endgroup$
    – Ben Reiniger
    Mar 13, 2020 at 3:14
  • $\begingroup$ 'caret' in R and 'scikit-learn' in Python, for example $\endgroup$
    – sgk
    Mar 13, 2020 at 15:22
  • $\begingroup$ I mean an example of how those packages can optimize the metrics you mention. $\endgroup$
    – Ben Reiniger
    Mar 13, 2020 at 15:38

1 Answer 1

0
$\begingroup$

Splitting is the same your are optimising for the parameters.

In other words, gini, entropy etc logic will stay the same but the breadth, width, number of features etc will be different when you optimise it with different loss functions.

Accuracy, precision, F-scores etc. are evaluation metrics computed from binary outcomes and binary predictions. They are NOT (but there are modification) for them to be a loss function. For model training, you need a function that compares a continuous score (your model output) with a binary outcome - like cross-entropy. Ideally, this is calibrated such that it is minimised if the predicted mean matches the population mean (given covariates)

$\endgroup$
1
  • $\begingroup$ Okay. So the tree is constructed using information gain, but to choose between different variations of the tree, to choose parameters to tune the tree, we use these metrics. That makes sense. $\endgroup$
    – sgk
    Mar 13, 2020 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.