1
$\begingroup$

For example, if you are a data scientist at a company, and a salesperson offers you a pre-trained model that is claimed to have a 90% accuracy of detecting fraudulent transactions. How would you go about verifying this claim?

I would test it on the data that I possess and see what the acc. is for detecting frauds in my data-set, and run a t-test with the sample acc., to see if it is close enough to 90% to make the call that it would detect 90% of all frauds in the entire population.

Let me know if this is the correct way to think about this question.

Thanks

$\endgroup$
  • $\begingroup$ I have no professional knowledge but I think the idea should work. We will take the model and test it on our dataset. If we get the desired validation accuracy on our dataset then we can buy the model. Also, the model needs to be tested on a larger dataset so as to avoid any problems or misleadings. $\endgroup$ – Shubham Panchal Feb 20 '19 at 5:27
1
$\begingroup$

There will already be a historical dataset of Fraudulent transactions (E.g.: If we take Credit Card as an example, all transactions that were disputed and dispute was accepted).

Sales team will include pre-sales developers that can help run a subset of such data through model (After various paperwork such as NDA, Legal review that a specific attribute like Gender/ Race can be used for this purpose). This will usually require :

  1. Extracting data from source systems or warehouse or data lake
  2. Transforming data for the model's input/output format
  3. Quantitative + Subjective validation by Operations team that handles such transactions

Getting 90% accuracy is pretty straight-forward. Since most of the transactions are not fraudulent, any model that predicts all transactions as genuine will meet this bar.

So step #3 (validation) will include agreement on appropriate measure for validation of the model (Such as F1 score)

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Good answer @Shamit, in addition to the above answer you can add more metrics for the step 3 like Sensitivity, Specificity along side of F1. $\endgroup$ – Toros91 Feb 20 '19 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.