1
$\begingroup$

I am collaborating on a research project with a respected econometrician as a graduate student (although only in an MS program, not PhD program mind you) exploring the properties and comparing the performance of new variation on the Best Subset Selection (aka All Subsets Regression) Algorithm he proposed which he calls EER (Estimated Exhaustive Regression).

My my main role in this collaborative research project is to expand the number of benchmark standard/straight forward variable selection procedures with which to compare EER's performance (in terms of how many true structural regressors it is able to select in terms of the standard performance metrics like TPR, TNR, PPV, etc) when run on each of a set of 260k randomly generated synthetic datasets via Monte Carlo Simulation. His original work compared EER's performance with Backward Elimination Stepwise Regression and with another variable selection method not worth explaining here.

So far, I have added LASSO Regression and Forward Selection Stepwise Regression (the latter just because it was straightforward to run an FS in R after finishing with the debugging process for my BE code). But I can't really think of a simple 4th method, I was considering Elastic Net, but that's just adding another Benchmark which is very similar to one already included which is what I already did before and that just seems underwhelming to me.

p.s. If anyone reading this thinks I should seriously consider also adding Elastic Net as its own, separate, 4th (or 5th assuming I get a good answer to this question I am able to implement) Benchmark Comparison method, and if you do think so, please make your argument explicit for me so I can consider it.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Train a decision tree with XGBoost, then select just the features that are used near the root. $\endgroup$
    – J_H
    Commented Dec 24, 2022 at 21:13
  • $\begingroup$ Oooo, coolio, I have not yet heard of that one, so I am excited to learn about it which is exactly why I asked this question in the first place. So, could you please point me to a solid resource of two on it, perhaps a textbook subsection, perhaps a paper, or perhaps a tutorial for how to implement it in R? I would be much obliged! $\endgroup$
    – Marlen
    Commented Dec 27, 2022 at 20:46
  • 1
    $\begingroup$ I have only used this package. It is well documented, and apparently there are bindings for sundry other languages like scala, java, julia, ruby, R. Consider viewing a tree viz. $\endgroup$
    – J_H
    Commented Dec 27, 2022 at 21:07
  • $\begingroup$ @J_H Oh wow, that paper (which isn't even the final published version) is from 2016, that explains why I am not familiar with it lol! I have been mostly learning from a combination of the 2017 edition of An Intro to Statistical Learning and the only 2013 edition of Applied Predictive Analytics. But since this is a legit research project testing out a new learning method, I should include one very new Benchmark method if I can! $\endgroup$
    – Marlen
    Commented Dec 27, 2022 at 21:39

1 Answer 1

Reset to default
0
$\begingroup$

I would advise against also running an Elastic Net Regression as an extra benchmark because it is likely to select either the same set of factors, or a perfect subset of the set of features which your LASSO Regression selects, so you won't gain any new selections with which to compare the results of this new EER algorithm you are trying to evaluate here.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ oh my goodness, that sounds right, don't know why I did not think of it myself! $\endgroup$
    – Marlen
    Commented Dec 30, 2022 at 3:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.