I am working with multiple machine learning methods (support vector regression, random forest regression, and knn regression through scikit-learn) and want to know how to determine which method is most appropriate to pursue given incrementally increasing my training sample by 10,000 records.
My training data size is about 80k records, and my test dataset is approximately 30k records. I start with 10k records to train each of my models because the system I am working in has very limited cores (and runs on a VM on a different continent) so I can't simply start a training session and walk away for a few hours.
As I increase the training size, I expect that my training validation scores (RMSE, MAPE) will decrease on the full training dataset, and my testing validation scores will increase.
What are the metrics that I should capture about each run in order to make informed decisions about the complexity, speed, and long term viability of these machine learning models?
(This post does a great job of explaining the general steps associated with the *entire* data science workflow. However, I am particularly interested in exploring what would be considered the ideal set of metrics that I should capture after having trained, tested, and validated a machine learning model.)