1
$\begingroup$

What I am doing:

I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format:

enter image description here

What I want to do:

I want to use SHAP TreeExplainer to interpret each prediction my model gives in terms of product attributes and user ids.

What I am getting:

My model is drawing all the conclusions based on product names and user ids, instead of product attributes and user ids.

What I tried:

I discovered that each product name has a unique combination of product attributes, i.e. by knowing the product attributes you can find its name. So my idea was to remove the product_name column, leaving only the attributes.

My reasoning was that restructuring the dataset in this way would lead to the interpretability that I wanted without any performance loss (since the product name doesn't add any new information).

What I got:

The model performance decreased a lot. Even with a great deal of hyperparameter tuning, I couldn't get near the performance I had when also using the product name.

What I think maybe going on:

  1. My dataset is too small for the model to learn with the product attributes (10k samples, 60 attributes).

or

  1. Maybe there are some attributes adding bias and screwing with my model ability to generalize, leading to an overfit.

I am a little skeptical about the number 2, seeing that my training loss also went up when I removed the product name.

My question:

So, how can I restructure my dataset? Does anybody have a clue why my model can't reach the same performance without using the product name? Any light or ideas on what I can try?

$\endgroup$
0
$\begingroup$

What may be happening is that your attribute predictors are weak predictors, they are noisy. Meaningful decision trees can't be made out of product attribute features by xgb.

When you are adding name as a predictor, xgb finds some signal wrt your target variable - rating and thus you get a better score. So your name plus attributes model may be performing better than attributes only model for this reason.

So if you from domain experience know product attributes are very weakly related to rating then you can conclude that this feature set of attributes is not going to help you make accurate predictions. Or instead of relying on d omain expertise, you can use correlation or relevant statistical tests to understand attributes relation to rating and if found that relationship is non existent or very weak you can conclude model isn't possible.

So may be add more relevant features if possible if you want to make a reasonably good model.

Regards Vik

$\endgroup$
0
$\begingroup$

Regarding your first question, no your dataset is not small to give you bad results, although adding more data would definitely help. I have worked with smaller datasets than yours.

The answer to your second question depends on weather you have performed feature selection/engineering. Usually the best type of feature selection is with the help of domain knowledge/expertise. Statistical feature selection techniques may or may not be accurate so it is usually advised to first use domain knowledge to eliminate features. If the performance does not improve then only use other techniques, albeit with a pinch of salt.

What you can do is remove the product id feature as id's usually do not contribute much to the model other than increasing the dimension. (I don't know what domain you are working on but if you think that product id's are important and help in predicting the target, then by all means keep the feature.)

Other than that, there are many things you can do to improve your model, some of which are (first of all create a baseline model where you do not perform feature selection and hyperparameter tuning. This will help you get a base score to compare to):-

1.) Use domain knowledge for feature engineering and see if you can reduce the dimensions by combining 2 or more numerical features into 1.

2.) Use domain knowledge for feature selection and if that doesn't help, use other feature selection techniques.

3.) Hyperparameter tuning

4.) Try different models

5.) Add more data

One or more of these things is absolutely going to help you in building a better model.

Remember, model building is an iterative process and you have to keep trying different things in order to get the best possible model. Don't try a couple things and come to the conclusion that the model is not the best one.

Best of luck!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.