I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By using sparse methods (LASSO, elastic net) we can remove several features to reduce the model.
My question is (theoretically): before we use metrics to assess the model selection are there any empirical observations which relate the optimal number of features to the number of observations?
For example: for a binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use?