Now obviously there is no such thing as an ideal number as every problem is different, but I've been Googling, ChatGPTing, & Youtubing this question for a few days now and I am constantly getting conflicting feedback. Some sources say you should throw as many features as you can engineer that are within reason to the problem at feature selection methods, and other sources say you should use domain knowledge and have a much more refined scope for the feature selection methods. I know this is a general question, but I'm hoping someone has a general "rule of thumb" response for this question. But if we are talking about specifics, I am working on a binary classification model using time series / balanced data. I could be at roughly 600 features for 6 months worth of data or 200 depending on the route I take.
3 Answers
Feature selections is a very conflicting topic as different people will have different opinions. Also what works on one dataset might not work at all for a different dataset.
IMHO, the best and the most accurate Feature Selection technique out there is Domain Knowledge. Use it to remove noise feature or create new useful ones. That way you are reducing the noise/bias/skewness in the most accurate/correct way.
That being said not everyone has the domain knowledge of every possible domain. In that case try to remove noise features or create new useful ones with the help of some common sense and Googling. You will be surprised by how much information you can get about a domain just by simple Googling.
If everything fails, then and only then use automated Feature Selection techniques. Probably use ML based techniques (like Forest or Boosting based importances etc).
This is just my opinion and preferance as this has worked for me pretty well, but many people might agree or disagree with me. You can try out multiple techniques and see which one works for you the best.
Cheers!
-
$\begingroup$ Thanks for that response, that explains why I'm finding so much conflicting information. I've been avoiding domain knowledge because it will take much longer to program, but hey if it was easy everyone would do it 😂 $\endgroup$ Commented Jul 10, 2023 at 6:43
I asked this question when I was still very new to Machine Learning, after a lot of practice and learning, I've been able to massively improve my models accuracy and I want to give an answer based on my findings. There is no such thing as too many features when using feature selection. However, it's important to make sure you identify the features that are just noise so you don't keep including them in your feature selection process. Domain knowledge, batching, and multiple feature selection methods really helped shine a light on those features that are just "noise" if you are lost.
The number of features you should use in your model depends on several factors, including the complexity of your problem, the amount of data you have, and the computational resources at your disposal. Few points that you could consider while choosing your features:
Overfitting vs. Underfitting: If you have too many features relative to the number of observations, your model may overfit the data, meaning it will perform well on the training data but poorly on new, unseen data. On the other hand, if you have too few features, your model may underfit the data, meaning it will not capture the complexity of the problem and will perform poorly even on the training data.
Curse of Dimensionality: As the number of features increases, the amount of data needed to generalize accurately grows exponentially. This is known as the curse of dimensionality. If you have numerous features but not much data, your model may struggle to learn.
Computational Resources: More features require more computational resources to process. If you're working with a large dataset and have limited computational resources, you may need to reduce the number of features.
Domain Knowledge: If you have a good understanding of the problem domain, you can use this knowledge to select the most relevant features. This can often lead to better performance than simply throwing in as many features as possible.
Feature Selection Techniques: Techniques such as mutual information, chi-square test, correlation coefficient, recursive feature elimination, etc., can help in selecting the most relevant features.
Model Complexity: Some models can handle high dimensional data better than others. For example, tree-based models like Random Forests and Gradient Boosting can handle high dimensional data quite well, while linear models may struggle. You can create a random forest model and select the top 50 or 100 features (depends on you about how many features that you need) from the model. This could work as well.
In your case, with time series data, it's important to consider the temporal aspect of your features. Some features may be highly correlated with each other due to the temporal nature of the data, which can lead to multicollinearity issues. Feature selection techniques that take into account the temporal aspect can be beneficial.
Finally, there's no thumb rule answer to this question. It's often a good idea to start with a larger set of features and then use feature selection techniques and cross-validation to get down the number of features to a manageable size.
