I am seeking theoretical suggestions more than else. This is my first actual practical work, and I am kinda stuck now that I have done a few things: I don't know in which direction to look in order to improve my model.
What kind of data I have
I have a dataset of information about a population. Each entry is a percentage of people belonging to a certain category (my features) over the whole number of inhabitants. I have
52 observation and
35 features, so a
52x35 input matrix. So my target is a
52 elements, and these are percentages too.
male female students employees freelancers unemployed catholics... city1 ##% ##% ##% ##%... city2 city3 ... city52
and my target vector is the results of the winning candidate of the last presidential election in each city.
What I did
I thought that, having such a small dataset, it would be better to stick to simple models, staying away from polynomial models etc. in order to avoid overfitting. Some engineering work to do on the features, too.
I removed redundant information and I agglomerated variables, for example by summing up
"irrelevant" percentage (compared to other values of the same conceptual categories) and filing them under new "other" features.
As I said, I decided to keep it simple, start with some simple linear regression and then checking how other regression models performed compared to linreg.
Dataset was splitted into
train $(30%)$ and test set. For each model I selected the related parameter which maximized the score. I then did
Random Feature Elimination (with cross validation) for each model, using the parameters selected in the previous step, leaving just a few most relevant features (since I know I have few observations compared to the number of features), and took this as final output.
I focused mainly on linear and ridge regression, with the latter performing better. My best
R2 is around
I see a bit of $overfitting$, but no clear path in the residuals. The target points with the worst predictions actually make sense in the context of the work (they are all belonging to a certain category).
After this I tried with some trees and randomforest regressors, but they performed worse than ridge regression.
What to do next
Well, this is where I lose my path. I think I did the right things until now, considering the kind of data I have (but please feel free to correct me), but.. is that it? Am I done or do I have something more to do, or maybe do again something in a different way? When can I call my self satisfied?
This is my very first ML work so any suggestion is very welcome.