Only data scientist in an organization and I could really use a sounding board here.
In Phase One of a project I deployed four models and served their average as the prediction. I used Random Forest regressor, AdaBoost regressor, Gradient Boosting regressor, and XGB regressor.
I was able to capture data on the back end for each individual model as well as the served prediction. It turned out that AdaBoost was giving the best results with respect to my metric of choice: mean absolute error.
In Phase One of this project I didn't perform any feature selection at all.
Phase two of the project is underway and I am attempting to get better predictions, and it is my presumption that performing feature selection will improve model performance.
I've got few feature selection methods I will be using: intuitive selection (using my intuition as to which features may be most important) as well as some filter methods (variance threshold and correlation measures) and feature importance from decision tree.
All in all I intend on using 4 feature selection methods.
Using the data that was used to train the model and scoring with on the data for which I already have ground truth I intend to retrain each of the models using the methods mentioned above.
So now that is 4 feature selection methods for 4 algorithms, for a total of 16 trained models.
The problem I have now is how to design the experiment to see which models perform better.
What would be a reasonable way to go about this?
My intuition says to take the model fitted by each feature selection method, compute the MAE, and do a pairwise t-test against the deployed model, testing a one-sided hypothesis that the MAE for the new model is less than the MAE in the deployed model.
Should I adjust for multiple comparisons?
Should I instead look to AIC, BIC, or Mallow' C? Are those applicable here? It seems tree-based models don't fit using maximum likelihood !