I am in the middle of a research project with a collaborator in which he has proposed a novel statistical learning processor for optimal variable selection, and I am running the 3 Benchmark Variable Selection Algorithms (LASSO, Backward Elimination Stepwise Regression, and Forward Selection Stepwise Regression) in RStudio on a set of 260,000 synthetic datasets the principal author has created via Monte Carlo methods. How well his new procedure performs is partially evaluated in comparison to the three benchmarks.
My problem is that I am just not able to run any of these Benchmark methods on the set of 260k synthetic data sets on my laptop without it taking days even though my laptop has 32 GB of RAM, a quad core i5 processor, and a nice big 1 tb SSD I put in it as well.
So, I am well aware of cloud computing services and parallel processing at a higher conceptual level, but I have never used AWS before. I just created an AWS account for myself, but the number of different service or "widget" options is extremely overwhelming at first. And even if I did find a suitable one, the whole way it works is very alien to me. Any tips, links, pointers, or advice will be much appreciated!
