1
$\begingroup$

I will evaluate classification models I made. That's logistic regression and decision tress.

1. What I use standards for comparison?
2. Suppose model selection's standard is ASE. One is high ASE of training data, low ASE of test data, and the other is ASE of training data is low and ASE of test data is high.. If you select a model, which models do you choose?

$\endgroup$
1
$\begingroup$
  • Accuracy (for classification problems)
  • Precision
  • Recall
  • F1 Score
  • AUC-ROC, particularly for imbalanced datasets

Good performance on the training set and bad performance on the test set is due to overfitting. So you should try to find ways to tackle overfitting, such as regularization of parameters, parameter tuning using cross-validation etc...

| improve this answer | |
$\endgroup$
0
$\begingroup$

The standards for evaluation of classification accuracy are:

  1. F-score (usually F1 score, giving same importance to precision and recall)
  2. AUC - ROC for binary classification (check this)
| improve this answer | |
$\endgroup$
0
$\begingroup$

In fact there is not any standards and it completely depends on your case.

If the subject of your problem can be ignored some other characteristics such as balance/imbalance and binary/multi-class can narrow your search results.

Besides, there are many benchmarks which evaluate the classification results according to many parameters such as data-set population, class-based accuracy, overall accuracy to name but a few. Some of these benchmarks are Landis-Koch Benchmark, Fleiss’ Benchmark, Altman’s Benchmark, Cicchetti’s Benchmark.

Disclaimer:

If you use python I suggest you to use PyCM which recommends the most fit metrics for evaluation and comparison. Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

>>> score = cm.Kappa
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.