Imbalanced classification with bias

Question

The problem:

A business historical heuristic rule for offering a special deal to customers has created a bias in the dataset when trying to use machine learning in order to make a more sophisticated customer selection for this particular deal.

In more detail:

for all customers that had a customer performance metric score > 90%, a deal was offered. A 10% of those has appeared to be a bad decision since they accepted the deal but failed to meet the requirements after. So with that information in hand, I labeled my data in the following way: every customer that accepted the deal but failed to meet the requirements is 0 and everyone else who accepted the deal is 1.

class 1: 90%
class 0: 10% 

However, even if it's a very intuitive way to label the classes, the heuristic that was in place created a bias in the data that is confirmed due to very poor baseline model performance. Fails completely to classify correctly that 10% of bad customers. Features used are the snapshot of basic important customer features at the day of the offer and model metrics used are AUC, Precision, Recall.

This is due to the fact that every customer had an exceptional customer performance metric score of 90%+ prior to the offer and it's the offer itself in many cases that changed the dynamics.

Is this a common scenario where heuristic logic is replaced by machine learning and the data is biased due to the simplistic approach of the heuristic rule in place?

Has anyone encountered a similar situation and how did you overcome this issue?

Two strategies to test(In parallel):

1. Extend the definition of my class beyond this specific deal, including customers that failed to meet requirements with other products and deals as well. - Eliminates the bias in the classes

2. Come up with features that can separate the two classes prior to the deal offer even though both groups seem identical from the customer performance metric standpoint. - Assumes that might be underlying features beyond customer performance metric that can differentiate the two types of customers

Answer 1

If the dataset has imbalance class (i.e. class 1 has 90% and class 0 has 10%) try to add some techniques like Up sampling or Down sampling in pre-processing stage to predict the sophisticated customer selection for that particular deal.

Answer 2

When you have an imbalanced class in a classification problem–the model tends to disregard the lower class sample. This is a problem because the models accuracy can be high but the errors for a single class would be very high. A good metric to look at would be the f1-score.

To fix an issue like this, you need to use SMOTE (synthetic minority oversampling technique) the library most commonly used it imblearn.

Answer 3

For imbalanced data there are some techniques that you can use to compensate imbalance namely:-

1.) Random Under sampling:- remove some observation from majority class to compensate for imbalance. Perform this only with huge amount of data.

2.) Random Over sampling:- add observations with some changes to minority class. perform this only with small data.

3.) SMOTE (Synthetic Minority Oversampling technique):- Synthesize new observations for minority class using KNN.

4.) NearMiss:- Unser sampling technique. Using a distance this will make the majority class equal to minority. Again uses KNN.

5.) Change the metric used:- Do no use accuracy for imbalanced data as it will lead you to think your model did good when in fact it performs poorly on real world data. F1 score and AUROC are your best bets

6.) Penalized algo's:- Penalizes mistakes on minority classes by an amount proportional to how under represented the minority class is. Examples are Penalized SVM.

7.) Change the algo:- Tree based models usually perform better on imbalanced data compared to other models. Examples are RandomForest, Gradient Boosted Trees.