Is this a good classified model based confusion matrix and classification report?

Question

I build a classification model. The results below when implemented Decision Tree Classifier.

Prediction accuracy

0.785813630042

confusion matrix

[[ 1302 1581]

[ 2577 13953]]

         precision    recall  f1-score   support

    2.0       0.34      0.45      0.39      2883
    3.0       0.90      0.84      0.87     16530

avg / total 0.81 0.79 0.80 19413

For the Neural Network---

Prediction accuracy

0.863132952145

Confusion Matrix

[[ 718 2165]

[ 492 16038]]

         precision    recall  f1-score   support

    2.0       0.59      0.25      0.35      2883
    3.0       0.88      0.97      0.92     16530

avg / total 0.84 0.86 0.84 19413

For the Gradient Boosting Classifier---

Prediction accuracy

0.870035543193

Confusion Matrix

[[ 971 1912]

[ 611 15919]]

         precision    recall  f1-score   support

    2.0       0.61      0.34      0.43      2883
    3.0       0.89      0.96      0.93     16530

avg / total 0.85 0.87 0.85 19413

Gradient boosting classifier gives me good accuracy score but it gives bad recall for class 2 compared with Decision Tree classifier. Train Data for both classes:

Class 2: 2908

Class 3: 16673

Is recall depend on training data? How do I improve recall for class 2 along with accuracy?

Answer 1

The question 'is this a good model?' is essentially a business question. There is always a precision/recall tradeoff and you decide based on your business goal what model to choose.

Data science can help you with the question on how to compare classifiers.

Let us start, that each (or nearly each) classifier can predict not only the outcome (Class2 or Class3), but also the probability of this outcome.

Your confusion matrix and all your metrics are suitable for making business decision, but, from data science point of view, have two problems

1) they do not take into account the agreement by chance

2) they are based on the threshold probability 0.5, which is not necessarily the optimal probability for deciding about the class.

To deal with (1), have a look at Cohen's kappa

To deal with (2), use metrics based on probability. The simpler one is Area Under the ROC Curve, see also here. Even finer, but not so straightforward to interpret is Logarithmic Loss

Is recall depend on training data? How do I improve recall for class 2 along with accuracy?

Recall depends on the quality of your model (and thus also on the quality of your training data) and on the chosen probability threshold. By decreasing probability threshold, you improve recall but worsen precision. Which tradeoff to choose is again a business decision.