I have a dataset, which contains the processing conditions (i.e., 42 features) and the property (i.e., 1 target) of a class of material. To know the performance of different machine learning models, I tested five different machine learning models by considering different numbers of features in training. These models are linear regression (LR), Bayesian ridge (BR) , k-nearest neighbor (NN), random forest (RF), and support vector machines (SVM) regression. The coefficient of determination (R2) for the test dataset is used to represent the performance of trained machine learning models.
As we can see that the max. accuracy of these models are in order: RF>BR~LR~SVM>NN. The top 8 features are required to obtain good accuracy for RF and afterward the accuracy is almost independent of the number of top-ranking features. The performance of BR, LR and SVM continuously improve with an increasing number of top-ranking features until reaching the maximum with top 26 features. NN exhibits different trends from others. It already reaches the best performance with top 8-12 features but gets worse and worse with more features.
I am wondering what are the possible reasons for this results? Some directions or some hints? for example:
- why the accuracy is in the order of RF>BR~LR~SVM>NN.
- why RF model reaches a good accuracy with a few features then keeps almost constant with more features?
- why linear-based models BR and LR have very similar performance as the SVM models.
- why NN model reaches a good accuracy with a few features then the accuracy reduces with increasing numbers of features?
I understand the reason is from case to case, but what is the general explanation or directions for finding the answers?