Number of rows vs. number of variables

Question

I am new to the ML/DS field. I started a project where I need to predict the age of users. I'd like to build two models, one predicting the age group (I created 6 age groups), and the other one predicting the age explicitly (as a number). I cleaned the data, and did the correlation between the possible input variables and target variables.
What I get confused about is: how many variables is too little compering to the dataset, and how many variables is too much?
Some people are suggesting that "the more the better", but I am also aware that too many features can cause overfitting...
Is there a rule of thumb? ex: "if I have a dataset of 500k it's reasonable to have 50 variables..." or is this really depending from case to case, and I should figure out variable importance to the model, once it is done?
Please let me know if something needs to be clarified, I'd really like if someone could help me with their experience and/or direct me to an article/paper that deals with this subject.

Answer 1

Up to my knowledge, no rigorous studies exist to give you an answer.

Also because the robustness of a model depends on it's ability to handle the overfit: a neural network without any regularization will be much less robust w.r.t one with regularization. And both will be less robust w.r.t. a Bayesian Neural Network.

So the Bayesian one will require much less samples in the dataset in order to give reliable results w.r.t. the non regularized Neural Network.

The robustness will also depend on the number of parameters on which your model is built on: a Neural Network with 50 Million of parameters need more data w.r.t. one with just one Million parameters.

So the better way to choose how many features to keep is to check if you are able to build a reliable model or not: start training your model with all the features; if the model gives you a good performance keeping the overfit under control, than you are ok with that. Otherwise you start pruning features.

Answer 2

There are general rules of thumb for different models, both in terms of the number of samples required as well as the number of features. I will try to just put some concrete numbers on things, with the caveat that is is indeed problem specific, and you usually have to make compromises.

If you are looking at something like a simple linear regression e.g. with 5 features, then for the basic tests to be considered statistically relevant, you need at least 40 samples.

In your case with 500,000 samples, 50 features is definitely acceptable - you could even have more. There are, however, two things I would be wary of:

Firstly, you make life hard for your model is many of the features are really describing the same thing. The technical term is multicollinearity, which, if your data has it, means that your features themselves are highly (linearly) correlated. It is hard for the model to know which feature to rely on, and your results may be improved by simply removing one. A simple example: if I predict your shoe size based on your height, and I have two features: your height in inches and your height in centimetres, you can see that I only really need one of these, as they are perfectly correlated measures.

Secondly, The more the better is theoretically true, as long as you have a model which is able to decide when to include features and when to ignore them - and that your data is good enough to facilitate this. There are ideas like Occam's razor, saying that a simple model should be selected over a complex model with the same performance; i.e. you should also consider interpretability. There are measures defined that quantify this, such as the Akaike Information criterion. It is a model selection tool - so you could fit the model using 10, 20, 30, and 50 of your features, compare the results of this creiterion and pick the model which scores best. Here is a recent question which touches on that topic.

There are functions that will remove features, based on a correlation threshold with other features in the dataset. It is explained well in the Caret package, for R, but the same thing can be done in Python fairly easily (not sure if there is a standard implementation). [EDIT] - the Python equivalent looks to be the VarianceThreshold class.