3
$\begingroup$

I am trying to assign costs to the confusion matrix. That is, in my problem, a FP does not have the same cost as a FN, so I want to assign to these cases a cost "x" so that the algorithm learns based on those costs.

I will explain my case a little more with an example:

  • When we want to detect credit card fraud, it does not have the same cost to predict that it is not fraud when in fact it is than the other way around. In the first case, the cost would be much higher.

What I wanted to know is if there is a library in R in which I can assign costs to these wrong decisions (i.e. give a cost to each possible value of the confusion matrix) or if there is an algorithm that learns based on a cost/benefit matrix.

I could also use some way to implement this without the use of a library.

Thank you very much.

$\endgroup$

2 Answers 2

2
+50
$\begingroup$

In this case, you can consider a payoff matrix to use average profit as your evaluation metric. It is important to mind the difference between the cost function (used in the learning process of the algorithm during training, which must be also differentiable with gradient-based optimization) and the performance metric (which is what I think you should consider in this case, your average profit for instance).

In case you have the following confusion matrix:

enter image description here

you can assign a benefit/loss per quadrant of your matrix:

enter image description here

and in this case, your evaluation metric (to select your best model):

$ Average Profit = \frac{(TN number*TN benefit + TP number*TP benefit - FN number*FN losses - FP number*FP losses)}{Number Of Predictions}$

This metric can also be used to select the optimal threshold of your model:

enter image description here

In case you want to feed you training algorithm to an XGB in R for instance, have a look at the documentation, where you have an argument called feval, to include your custom evaluation metric function (documentation link):

feval --> customized evaluation function. Returns list(metric='metric-name',value='metric-value') with given prediction and dtrain

$\endgroup$
2
  • $\begingroup$ Hi, the explanation is clear to me, what I am looking for is how to implement this idea in R so that the algorithm learns based on these costs. $\endgroup$
    – PicaR
    Jun 2 at 6:49
  • 1
    $\begingroup$ once you have defined the custom evaluation metric (it could be average profit), you can pass it as argument to the "fit" (or similar) function, depending on the library you use. In case of XGBoost in R, you can have a look at the documentation, where it mentions feval argument for you custom evaluation metric function (I will also update it in my answer) $\endgroup$
    – German C M
    Jun 2 at 7:41
2
$\begingroup$

What defines the penalty value of each FP or FN? It is the final cost due to those decisions compared to the TP and TN.

Having a false positive of 1M dollars of credit card fraud, has not the same importance than having a false positive of 10 dollars.

That's why the cost function shall be directly linked to the amount of the frauds, and hence the objective function of your algorithm.

Therefore you can multiply each decision result (TP,TN,FP,FN) by the amount of each payment: Remember that you aim to improve the whole model, including the right result, not just the bad ones.

Consequently, the objective function is to try to find the maximum of the sum of all payments, taking the good payments (TP,TN) as positive, divided by all the amounts (TP,TN,FP,FN).

For exemple:

TP= [2500, 200, 60]
TN= [300,5000]
FP= [1500,30,600]
FN= [300, 200]

Score = (2500+200+60+300+5000) / (2500+200+60+300+5000 + 1500 + 30 +600 +300 +200)

Score = 0.75

Which means that 75% of the total money has been correctly assigned, and the aim is to reach 100% as close as possible.

If you want to take into account the quantity of frauds as well, you just have to add and artificial penalty (let's say 100 dollars) to each transaction, so that you can also value the quantity of frauds, instead of the final raw cost result.

Score = ( 2600+300+160+400+5100) / (2600+300+160+400+5100 + 1600 + 130 +700 +400 +300)

Score = 0.73

This option is usefull when trying to reduce the importance of big transactions, in favor of the small ones (= considering more the small customers).

You can find more information about cost sensitive cost function here: https://towardsdatascience.com/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9

$\endgroup$
5
  • $\begingroup$ correct me if I misunderstood, but I think you are mixing the concepts cost function (used in the training process by the learning algorithm) and evaluation metric (which takes into account TP, FP, FN, TN, and its corresponding benefit/loss values) $\endgroup$
    – German C M
    Jun 1 at 9:18
  • $\begingroup$ Why are they neccesarily dissociated? The aim of a cost function is to improve an algorithm with reliable indicators, and what is the most reliable indicator in this case? Otherwise, how would you build a reliable cost function? $\endgroup$ Jun 1 at 18:28
  • $\begingroup$ Your cost function (for usual gradient based optimization) must be a (mathematically) differentiable function. The indicator you mean is rather the evaluation metric, which sometimes can be the same as the cost function (RMSE in regression for instance) but not always, since this indicator is a metric used after the algorithm learns using the cost function, but it is not the same. $\endgroup$
    – German C M
    Jun 1 at 19:51
  • $\begingroup$ Hi, the explanation is clear to me, what I am looking for is how to implement this idea in R so that the algorithm learns based on these costs. $\endgroup$
    – PicaR
    Jun 2 at 6:49
  • $\begingroup$ It depends on the algorithm you are using. I've added a link that explains how to implement a cost sensitive cost function. In addition to that, R seems more flexible than Python regarding model customization (for instance: stackoverflow.com/questions/42372821/…) $\endgroup$ Jun 2 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.