Python - Calculate Cost profitability and benefit of the model

Question

I've this code in Python in order to calculate the precision of my model and to print confusion matrix using Decision Trees Classifier:

coef_gini = DecisionTreeClassifier(criterion = "gini", random_state = 100, max_depth = 3, min_samples_leaf = 5)
coef_gini.fit(training_features, training_target)

y_pred = coef_gini.predict(test_features)
y_pred

for name, importance in zip(training_features.columns, coef_gini.feature_importances_):
    print(name, importance)

print ( "Train Accuracy using Decision Trees Classifier is : ", accuracy_score(training_target, coef_gini.predict(training_features)))
print ( "Test Accuracy using Decision Trees Classifier is : ", accuracy_score(test_target, y_pred))
print ( "Confusion matrix using Decision Trees Classifier is ", confusion_matrix(test_target, y_pred))

What is the cost matrix? Is this the money that company will lost for each wrong predictive target value? Anyone have an example?

Thanks!

Answer 1

Confusion Matrix

A Confusion Matrix is an important tool to measure accuracy of a classification algorithm. It compares predicted class of an outcome and actual outcome.

Scenario 1: Credit Risk

Based on a credit risk scorecard, application for credit card are classified as “Good” and “Bad”. “Good” indicates applicants paying back dues on credit card and “Bad” indicates customers defaulting on the dues. Now, the customers are compared against actual performance of the customer payment behaviour after say 18 months. So, comparison of Predicted Class (“Good” or “Bad”) to actual customer behaviour state (“Defaulted” or “Regular”).

There is always a trade-off between Type I error/False Positives (accepting Bad Customers) and Type II Error/False Negatives (Rejecting Good Customers).

We generally require to optimize between False Positive Rate (Type I Error) and False Negative Rate (Type II Error).

So, role of cost matrix comes in picture to find the optimal cut off value for a classification rule.  Now, going back to Credit Risk Model. The cut off value optimize between cost of an opportunity loss (miss to accept a good customer/Type II Error) and cost of accepting a potential defaulter (involved in loss due to default).

Referencing Example is from a blog and image from Google..

Cost Matrix

Cost Matrix is similar of confusion matrix. It’s just, we are here more concerned about false positives and false negatives .There is no cost penalty associated with True Positive and True Negatives as they are correctly identified.

The goal of this method is to choose a classifier with lowest total cost.

• Total Cost = C(FN)xFN + C(FP)xFP

where,

• FN is the number of positive observations wrongly predicted

• FP is the number of negative examples wrongly predicted

• C(FN) and C(FP) corresponds to the costs associated with False Negative and False Positive respectively.

  Remember, C(FN) > C(FP).

Hope this Helps..