I have a machine learning problem with about 160 features and 400 cases and I want to find the best predictors for a continuous outcome. The dataset contains variables of psychotherapists and clients. I want to predict therapy outcome.
I used lasso regression in nested 20-fold cross-validation and could identify about 20 top predictors (model fit about 0.97 nrmse). (I decided not to create a seperate holdout dataset, because I have too few cases.) However, I thought I could improve model performance with xgboost, but even though I used GridSearch (colsample_bytree=0.3, learning_rate=0.01, max_depth=2, n_estimators=1000), I did not manage to get to this fit (model fit about 1.01 nrmse). Is xgboost overfitting? Right now, I am not sure how I can improve model performance. Do I

  1. need to use a dimension reduction technique (possibly pca?) before I employ ML?
  2. select the lasso top predictors as my new prediction features?
  3. use a different algorithm (possibly knn or svm?) on either 1. or 2. ?

Btw, I use the shap framework in python to assess feature importance. Therefore I am not bound to any algorithm to identify the best features.

Thanks for any help in advance!

  • $\begingroup$ Welcome to DataScienceSE. I'm not very familiar with NRMSE but if I'm not mistaken a value close to 1 is seriously bad performance. A value of 1.01 is very strange, since it's supposed to be lower than 1. At first sight it looks like either there's an error in measuring NRMSE or the data is completely useless (but maybe I'm missing something). $\endgroup$
    – Erwan
    Commented Mar 27, 2022 at 0:00
  • $\begingroup$ I guess that depends on how rmse is normalized, which was by dividing by standard deviation. So this of course means the model is far from good, but it would mean a lot worse fit if I had divided by maxmin. $\endgroup$ Commented Mar 28, 2022 at 8:01
  • $\begingroup$ Oh ok. About overfitting it's simple enough to test: apply the model on both the training and test set, if there is a really high difference in performance between the two then there is overfitting. The rest of your questions I don't know, but keep in mind that it might not be possible to improve performance on this dataset. $\endgroup$
    – Erwan
    Commented Mar 28, 2022 at 15:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.