0
$\begingroup$

I've been stuck on the following problem for weeks now. To be clear I'm not asking the community to provide a full solution. Just a few ideas or at least confirmation on whether this problem statement is solvable with the data available at hand. Thanks.

Context: I work for a company that rents out stuff. During user checkout, there's a rule-based risk engine that determines whether to auto-approve user (takes seconds) or send for manual approval (takes days). This engine utilizes only basic features such as credit score, rent items category, cart value etc. Lets say that around 40% of all instances go to manual approval. Actual rejection rate is around 1-2% that means most of the cases sent for manual approval could have just been auto-approved.

Objective: Reduce dependency on manual approval as much as possible (to lets say 5%). If this was a binary classification task (manual or not) then we'd say recall is important here. Thus, deprecate legacy system as it is too rudimentary.

Now, boss wants me to achieve this by creating a regression model using customer credit history, and purchase history to predict maximum allowed †active-sourcing-cost per user. So that if a user tries checking out cart that brings their total active sourcing cost above their max allowed limit, they will be sent for manual approval instead of auto-approve.

†active-sourcing-cost is the total purchase cost to company of all items the user currently is renting

Data available: User purchase history, user credit history (experian), product details, NPA data (data of users who failed to return the items they have rented, thus defaulted)

Issues with proposed solution:

  1. Max allowed sourcing cost per user unavailable as the target variable (No recorded ground truth for any samples)
  2. No relevant proxy variables available as ground truth (that I can think of)

Tried explaining that lack of relevant ground truth is an issue. But am being pushed for a solution so have to produce.

My alternate half-baked solution:

  1. Use legacy system for auto approvals only. This acts as a filter.
  2. To the rest of the data that would otherwise be sent for manual verification we add NPA users and tag as a negative class. This would be the training data.
  3. Train binary classification model (npa vs. not npa).
  4. What the model classifies as NPA would then be sent for manual, rest will be auto approved.
Improve this question
$\endgroup$
3
  • $\begingroup$ I presume you already tried this, but what does EDA say about maximum allowed? You can't take 90% quantile of purchase made by users in past as maximum allowed? I would take business into confidence if that value makes sense from the business point of view and logically. $\endgroup$
    – monte
    Commented Jul 23, 2023 at 13:23
  • 1
    $\begingroup$ To my understanding, the real question is either, whether to approve someone or - even more accurate - whether someone defaults. Both questions would lead in a similar direction as your alternate solution. $\endgroup$
    – Broele
    Commented Jul 23, 2023 at 14:38
  • 2
    $\begingroup$ What I would change: build a model with continuous output (like predict_proba in sklearn). Then you can build a policy / rule closely along your business expertise to decide between automatic approval and manual processing. This can be done purely based on the probability (e.g. in a way, that only 5% are processed manually), but you could also include expected loss (risk-probability x active-sourcing-cost) into such a policy. $\endgroup$
    – Broele
    Commented Jul 23, 2023 at 14:42

2 Answers 2

Reset to default
1
$\begingroup$

What your boss is telling you to do seems bassackwards to me given the clear business problem you've laid out (40% goes to manual approval and only 2% of those are rejected). What I really mean here is a regression model doesn't make sense here given what you want to accomplish.

At the end of the day both you and your boss want to make a decision boundary for sending stuff to manual approval vs. auto-approval. He probably has some practical reason for wanting that decision boundary to be based on active-sourcing-cost, conditioned on your other inputs.

Do you have rental and return dates? If so, you could generate the active sourcing cost, right? Then you might be able to do something reasonable and make your boss happy (with much data massaging that is). If not, you have a pretty good data set for making a binary classifier that should reduce labor and turnaround time for rental approvals.

One thought on your NPA tags though. Surely there must be sometimes the equipment isn't returned and the rental went through auto-approval. If so, you should make use of that data as well (if you can).

Improve this answer
$\endgroup$
0
$\begingroup$

It's even weird that you would use the old model tag as a target. Seems like some sort of complex knowledge distillation. Imo you should start by just statistical learning on default. starting with a simple model, then iterating to deal with the different problems (Feature engineering / imbalance / calibration in probability / upgrading to xgboost / explainability / eventual interface to exaplin the choice / documentation / evaluate out of sample / evaluate the gain to upgrade the old model...). The proportion of the limit used would make a good feature for exemple.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.