As datasets and the number of parameters get larger, it becomes increasingly difficult to run validation locally because of limited disk size and computing power. As a result, people might use an old laptop or a server in the cloud. In particular, I'm interested in learning more about the second option.
For the setup, it sounds like I will need a storage system and a computing system. Is this S3 and EC2? Redshift and Amazon ML? Combinations from some other vendor? At the moment, I'm asking as an amateur participant for Kaggle, but I'd also be curious to know what the pros use.
On the machine itself, is there any way to interact with a GUI rather than a command line to set this up? Would I have to install Anaconda or some other Python distribution before getting started? Is there a specific file structure to use? What are the main pitfalls to watch out for?
Ultimately, I am looking for the practical advice to set everything up. If you're able to just offer links to documentation, that would be immensely helpful as well. Thanks!