I've been stuck on the following problem for weeks now. To be clear I'm not asking the community to provide a full solution. Just a few ideas or at least confirmation on whether this problem statement is solvable with the data available at hand. Thanks.
Context: I work for a company that rents out stuff. During user checkout, there's a rule-based risk engine that determines whether to auto-approve user (takes seconds) or send for manual approval (takes days). This engine utilizes only basic features such as credit score, rent items category, cart value etc. Lets say that around 40% of all instances go to manual approval. Actual rejection rate is around 1-2% that means most of the cases sent for manual approval could have just been auto-approved.
Objective: Reduce dependency on manual approval as much as possible (to lets say 5%). If this was a binary classification task (manual or not) then we'd say recall is important here. Thus, deprecate legacy system as it is too rudimentary.
Now, boss wants me to achieve this by creating a regression model using customer credit history, and purchase history to predict maximum allowed †active-sourcing-cost per user. So that if a user tries checking out cart that brings their total active sourcing cost above their max allowed limit, they will be sent for manual approval instead of auto-approve.
†active-sourcing-cost is the total purchase cost to company of all items the user currently is renting
Data available: User purchase history, user credit history (experian), product details, NPA data (data of users who failed to return the items they have rented, thus defaulted)
Issues with proposed solution:
- Max allowed sourcing cost per user unavailable as the target variable (No recorded ground truth for any samples)
- No relevant proxy variables available as ground truth (that I can think of)
Tried explaining that lack of relevant ground truth is an issue. But am being pushed for a solution so have to produce.
My alternate half-baked solution:
- Use legacy system for auto approvals only. This acts as a filter.
- To the rest of the data that would otherwise be sent for manual verification we add NPA users and tag as a negative class. This would be the training data.
- Train binary classification model (npa vs. not npa).
- What the model classifies as NPA would then be sent for manual, rest will be auto approved.