I understand how a decision tree is constructed (in the ID3 algorithm) using criterion like entropy, gini index, variance reduction. But the formulae for these criteria do not care about optimization metrics like accuracy, recall, AUC, kappa, f1-score and others.

Packages on R and Python allow me to optimize for such metrics when I construct a decision tree. What do they do differently for each of these metrics? Where does the change happen?

Is there a pattern to how these changes are done for different classification/regression algorithms?

  • $\begingroup$ Could you provide an example of packages optimizing such metrics? $\endgroup$ – Ben Reiniger Mar 13 '20 at 3:14
  • $\begingroup$ 'caret' in R and 'scikit-learn' in Python, for example $\endgroup$ – sgk Mar 13 '20 at 15:22
  • $\begingroup$ I mean an example of how those packages can optimize the metrics you mention. $\endgroup$ – Ben Reiniger Mar 13 '20 at 15:38

Splitting is the same your are optimising for the parameters.

In other words, gini, entropy etc logic will stay the same but the breadth, width, number of features etc will be different when you optimise it with different loss functions.

Accuracy, precision, F-scores etc. are evaluation metrics computed from binary outcomes and binary predictions. They are NOT (but there are modification) for them to be a loss function. For model training, you need a function that compares a continuous score (your model output) with a binary outcome - like cross-entropy. Ideally, this is calibrated such that it is minimised if the predicted mean matches the population mean (given covariates)

  • $\begingroup$ Okay. So the tree is constructed using information gain, but to choose between different variations of the tree, to choose parameters to tune the tree, we use these metrics. That makes sense. $\endgroup$ – sgk Mar 13 '20 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.