How can I improve my regression model?

Question

I am seeking theoretical suggestions more than else. This is my first actual practical work, and I am kinda stuck now that I have done a few things: I don't know in which direction to look in order to improve my model.

What kind of data I have

I have a dataset of information about a population. Each entry is a percentage of people belonging to a certain category (my features) over the whole number of inhabitants. I have 52 observation and 35 features, so a 52x35 input matrix. So my target is a 1D-vector of 52 elements, and these are percentages too.

       male  female  students  employees  freelancers  unemployed  catholics...
city1  ##%  ##%     ##%     ##%...
city2
city3
...
city52 

and my target vector is the results of the winning candidate of the last presidential election in each city.

What I did

I thought that, having such a small dataset, it would be better to stick to simple models, staying away from polynomial models etc. in order to avoid overfitting. Some engineering work to do on the features, too.

Feature selection

I removed redundant information and I agglomerated variables, for example by summing up "irrelevant" percentage (compared to other values of the same conceptual categories) and filing them under new "other" features.

Model selection

As I said, I decided to keep it simple, start with some simple linear regression and then checking how other regression models performed compared to linreg.

Dataset was splitted into train $(30%)$ and test set. For each model I selected the related parameter which maximized the score. I then did Random Feature Elimination (with cross validation) for each model, using the parameters selected in the previous step, leaving just a few most relevant features (since I know I have few observations compared to the number of features), and took this as final output.

I focused mainly on linear and ridge regression, with the latter performing better. My best R2 is around 0.7xx.

I see a bit of $overfitting$, but no clear path in the residuals. The target points with the worst predictions actually make sense in the context of the work (they are all belonging to a certain category).

After this I tried with some trees and randomforest regressors, but they performed worse than ridge regression.

What to do next

Well, this is where I lose my path. I think I did the right things until now, considering the kind of data I have (but please feel free to correct me), but.. is that it? Am I done or do I have something more to do, or maybe do again something in a different way? When can I call my self satisfied?

This is my very first ML work so any suggestion is very welcome.

Answer 1

I have a similar dataset atm and have been doing some research, not yet finished.

What needs to be kept in mind is the need for generalization given that there is such few samples and many dimensions.

The linear regression model is not robust enough given the dataset, it does not handle a high number of dimensions, compared to samples, well. Even if you could tune in order to get a good result, you are still probably overfitting.

Stepwise feature/model selectors such as Cp-statistics does not seem to do well with so many dimensions vs samples.

Ridge, Lasso, and Elastic net seems to be the way to go for feature selection. These penalized linear regression models handles the high number of dimensions vs samples much better. The linear regression is penalized in order to generalize better and not overfit, creating a less complex solution. R: GLMNET - Penalized regression Depending on alpha, you may receive a different feature set, coefficient that are not zero. E.g Lasso is biased for not returning features that are very correlated with each other while ridge might return a set of variables/dimensions/predictors/features that are correlated with each other.

K-Fold Cross-validation could be used in order to check the features selected are selected over and over again. This could be used in order to make sure that you don't overfit.

I hope this gives you a few ideas.