A business historical heuristic rule for offering a special deal to customers has created a bias in the dataset when trying to use machine learning in order to make a more sophisticated customer selection for this particular deal.
In more detail:
for all customers that had a
customer performance metric score > 90%, a deal was offered. A 10% of those has appeared to be a bad decision since they accepted the deal but failed to meet the requirements after. So with that information in hand, I labeled my data in the following way: every customer that accepted the deal but failed to meet the requirements is 0 and everyone else who accepted the deal is 1.
class 1: 90% class 0: 10%
However, even if it's a very intuitive way to label the classes, the heuristic that was in place created a bias in the data that is confirmed due to very poor baseline model performance. Fails completely to classify correctly that 10% of bad customers. Features used are the snapshot of basic important customer features at the day of the offer and model metrics used are AUC, Precision, Recall.
This is due to the fact that every customer had an exceptional
customer performance metric score of 90%+ prior to the offer and it's the offer itself in many cases that changed the dynamics.
Is this a common scenario where heuristic logic is replaced by machine learning and the data is biased due to the simplistic approach of the heuristic rule in place?
Has anyone encountered a similar situation and how did you overcome this issue?
Two strategies to test(In parallel):
Extend the definition of my class beyond this specific deal, including customers that failed to meet requirements with other products and deals as well. - Eliminates the bias in the classes
Come up with features that can separate the two classes prior to the deal offer even though both groups seem identical from the
customer performance metricstandpoint. - Assumes that might be underlying features beyond
customer performance metricthat can differentiate the two types of customers