What defines the penalty value of each FP or FN? It is the final cost due to those decisions compared to the TP and TN.
Having a false positive of 1M dollars of credit card fraud, has not the same importance than having a false positive of 10 dollars.
That's why the cost function shall be directly linked to the amount of the frauds, and hence the objective function of your algorithm.
Therefore you can multiply each decision result (TP,TN,FP,FN) by the amount of each payment: Remember that you aim to improve the whole model, including the right result, not just the bad ones.
Consequently, the objective function is to try to find the maximum of the sum of all payments, taking the good payments (TP,TN) as positive, divided by all the amounts (TP,TN,FP,FN).
TP= [2500, 200, 60]
FN= [300, 200]
Score = (2500+200+60+300+5000) / (2500+200+60+300+5000 + 1500 + 30 +600 +300 +200)
Score = 0.75
Which means that 75% of the total money has been correctly assigned, and the aim is to reach 100% as close as possible.
If you want to take into account the quantity of frauds as well, you just have to add and artificial penalty (let's say 100 dollars) to each transaction, so that you can also value the quantity of frauds, instead of the final raw cost result.
Score = ( 2600+300+160+400+5100) / (2600+300+160+400+5100 + 1600 + 130 +700 +400 +300)
Score = 0.73
This option is usefull when trying to reduce the importance of big transactions, in favor of the small ones (= considering more the small customers).
You can find more information about cost sensitive cost function here: