1
$\begingroup$

I am trying to work with Classification model. I am planning to train and test my model with a large dataset (Training with 80% and testing with 20% volume - no under/oversampling).

What I understand about classification models is that it focuses on a binary TRUE/FALSE or YES/NO type of prediction. However, I should also be using some evaluation metrics e.g. Accuracy/Precision/Recall etc. Ideally, there would be a threshold which I need to elevate/lower to meet my expectations.

My question - if I am to use any bespoke software solution (e.g. some Python/R library) for this, what am I expecting to see from my model to output? Is it as simple as YES/NO output or with some form of accuracy/precision number?

$\endgroup$
  • $\begingroup$ I'd recommend reading an introductory books so that you can know basic metrics and when to use them (introduction to statistical learning if I had to recommend only one, which is free online). Regarding implementation : depends on your model, implementation, see their specific documentation. $\endgroup$ – lcrmorin Nov 4 '20 at 12:27
0
$\begingroup$

Binary problems have the most amount of metrics to measure its performance. They can be classified as accuracy metrics, probabilistic metrics and metrics depending on true/false positives/negatives. You are not expected to implement them from scratch, the last time I implemented an AUC (area under the curve) was in college so you will find these metrics already implemented in most machine learning libraries in all languages. This is a comprehensive list taken from the Keras library, check the links for more information about the metrics or the library.

About the matter of the outputs you can have a true/false 1/0 values or the probabilities of the observation to belong to a class which is a better way to measure the algorithm's performance. Do you need a threshold to evaluate results properly, it all depends on the data you have. For some problems high accuracy like above 99.999% is critical like clinical diagnoses but for others accuracy like 55% can be seen as great like the case of stock predictions. So it depends on your data.

Accuracy metrics: These metrics, simple in design, are often raw percentages of the correctly labeled classes for one class or many classes.

Raw Accuracy

Binary Accuracy

Categorical Accuracy - Straightforward accuracy for all classes.

Top K Categorical Accuracy - Accuracy for the top K categories

Sparse Top K Categorical Accuracy

True False Positive Negative metrics: More expressive metrics useful to determine the impact the algorithm has not only in its accuracy but also in the wrongly classified observations.

AUC

Precision

Recall

True Positives

True Negatives

False Positives

False Negatives

Precision At Recall

Sensitivity At Specificity

Specificity At Sensitivity

Probabilistic metrics: These as often used as the function loss for machine learning algorithms, however, they can also be understood as a benchmark metric specially for a validation set.

Binary Crossentropy

Categorical Crossentropy

Sparse Categorical Crossentropy

KLDivergence

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.