0
$\begingroup$

Summary:

I am training a discount eligibility model where the dataset represents historical data for products where people availed discounts based on simple features like product category, discount code, amount, date of purchase, state etc. In real world scenerios most people enter the product and discount information correctly so the dataset I have consists of discounts applicable (class 0) 200 times more than discount not applicable (class 1) e.g. (label class 0/class 1 = 200/1) I have tried the following methods to handle this imbalance:

  1. Providing class weights to the model: Using the training parameters in documentation I set the "class weight" attribute to 200, 300, 400, 600 for label class 1. This results in high accuracy but poor F1 and Recall score < 50%.
  2. Using undersampling for class 1: Such that label class 0/class 1 = 1/1 the model gives good recall, accuracy and F1 score. But due to the undersampling I'm missing a lot of combinations/categories values in training. Catboost tries to estimate these unseen categories and gives predicts them to be class 1 when they should be class 0 with a prediction score of > 95%. i.e. the model says that 95% chance is the discount is not valid (belongs to class 1) but in reality it belongs to class 0 (valid discount).

What are the possible ways to handle such scenerios in random forest/decision tress based models when we want the model to learn all the possible categories to avoid pivot estimation but still give good recall and not become biased ?

Notes:

  • The cardinality of categorical columns used is > 500.
  • The training data is imbalanced but for testing I have created a sub sample of records where the class 0/ class 1 = 1. Also it includes records where a change in feature value results in discount applicable. i.e. User entered the info with state A the discount got rejected (class 1). When they entered state B the same record got discount applicable (class 0).
  • Both of these records exist in test data to correctly evaluate the performance of the model.

Detailed Scenario:

Product Tag 1 Product Tag 2 Product Tag 3 Buyer State Buyer City Discount Code 1 Discount Code 2 Product Code Purchase Date Discount Denied
Electronics Lights Decoration Texas Austin 3488833 222221 SKU388d 1/8/2023 0
Food Kitchen Dairy Texas Austin 323423   SKUz7738 1/8/2023 1
Kids Toys Puzzle Texas Austin 3232323   SKUkkkl9 1/8/2023 0
Music Digital   Texas Austin     SKU119dj 1/8/2023 1
Decoration Woodwork   Texas Austin 3488833 222221 SKUk47dj3d 1/8/2023 0

The last column "Discount Denied" is the label or class to predict in this case study. When I train the model with a balanced dataset i.e. equal numbers of samples for 0, 1 classes the prediction results perform good on the dataset for with Stats and Confusion Matrix

  • Recall: 0.7975513773502405

  • F1-Score: 0.7642991829038341

  • Accuracy: 0.7381894344891785

  • Confusion Matrix: [[1348 662] [ 463 1824]]

The model is good at detecting class-1 (discount denials). But for the false positive and false negative scenerios I ran some analysis. On investigation I found that test data included some combinations of features which were not present in data so the model was estimating the categorical features values for unseen data.

To overcome this I increased the size of dataset to include maximum possible products, discount codes and other feature combinations. By doing this I have increased the imbalance of label class i.e. The training data now has 200x class-0 labels. After passing "class weights" based on this ration 200, 300, 400 the resulting model produced the following stats:

  • Recall: 0.4209100758396533
  • F1-Score: 0.44668008048289737
  • Accuracy: 0.478602383531961
  • Confusion Matrix: [[ 990 856] [1069 777]]

Please guide about different techniques or similar case studies where the class imbalance is of such high rate but undersampling is not an option because we lose a lot of information. Is there any approach where these two models can be combined to build a better model ? Or maybe a mix of random-forest, decision tress and some other model can help solve such challanges.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.