What I am doing:
I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format:
What I want to do:
I want to use SHAP TreeExplainer to interpret each prediction my model gives in terms of product attributes and user ids.
What I am getting:
My model is drawing all the conclusions based on product names and user ids, instead of product attributes and user ids.
What I tried:
I discovered that each product name has a unique combination of product attributes, i.e. by knowing the product attributes you can find its name. So my idea was to remove the product_name
column, leaving only the attributes.
My reasoning was that restructuring the dataset in this way would lead to the interpretability that I wanted without any performance loss (since the product name doesn't add any new information).
What I got:
The model performance decreased a lot. Even with a great deal of hyperparameter tuning, I couldn't get near the performance I had when also using the product name.
What I think maybe going on:
- My dataset is too small for the model to learn with the product attributes (10k samples, 60 attributes).
or
- Maybe there are some attributes adding bias and screwing with my model ability to generalize, leading to an overfit.
I am a little skeptical about the number 2, seeing that my training loss also went up when I removed the product name.
My question:
So, how can I restructure my dataset? Does anybody have a clue why my model can't reach the same performance without using the product name? Any light or ideas on what I can try?