I have a training data set of 500 people and 26 features and I'm trying to develop a regression model. A possibility is to derive more features of course. I'm considering the following models:

  • Linear regression
  • Stepwise regression
  • Lasso regression
  • Ridge regression
  • Principal Component Analysis
  • Factor Analysis

The final goal is to find the model with the best generalization performance (results on predicting test set). Are there any guidelines/common practices for when to use which model (based on e.g. sample size, number of features)? Or should I just use cross-validation and choose the model with the lowest error?

My features look like: Features 1-13

Features 14-26

My target variable looks like: Target variable

Descriptive stats features: enter image description here

enter image description here

Descriptive stats y: enter image description here

  • 1
    $\begingroup$ We need more information on the data you have. Can you describe the features a bit maybe share a few rows of them? $\endgroup$
    – serali
    Commented Oct 1, 2021 at 10:37
  • $\begingroup$ What are the features? Is it something you can share? I meant what each column represents here. $\endgroup$
    – serali
    Commented Oct 1, 2021 at 12:01
  • $\begingroup$ How about statistical data of the colums? min, max average etc. $\endgroup$
    – serali
    Commented Oct 1, 2021 at 13:19

2 Answers 2


The predictive power of a model is highly contingent on the data generating process and it is ex ante hard to tell what will work best (especially with limited information about the data as in this case).

Lasso, Ridge, and Principal Component Analysis are often used to reduce dimensionality or to select features which is not a real issue in your case. Linear regression will work well but will likely not deliver good predictive power as it is a parametric technique.

Often random forest or boosting will work well for several reasons. First, you do not need to care about model parameterization. Second, you do not need to care about possible issues with multicollinearity in the data (a possible issue in linear regression). Third, you can investigate the feature importance from this models to see which features work well (and remove features with little or no predictive power).

My suggestion: Start with a linear regression as baseline model, then try random forest and/or boosting (e.g. using xgboost, lightgbm) and compare results. Maybe have a look at Lasso/Ridge as well. Use 5-fold CV to assess model performance. Maybe also see if feature generation helps, e.g. by running a random forest with only few terminal nodes (3-4 splits) on single generated features like $y = x_1 - x_2$, $y = x_1 / x_2$ etc. to see which feature interactions might have high predictive power. Add those generated features which perform best to the overall model and see if the model's predictive power is better then before.


As for the data, they all seem to be well distributed within themselves, so scaling should work well, and definitely must be done. As it is, they have very distinct mean values and any model would fail without normalization.

As for y values, even though values for 25%, 50% and 75% are relatively close to each other, max and min values are not, so you would be better off if you got rid of those outliers before applying any model. Normalizing with them included would not work very well. By the looks of the image, removing the 3 highest values should be enough.

As for the model, I have nothing to add on top of what @Peter suggested before.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.