Workflow for Iteratively Capturing and Improving Machine Learning Models

Question

I am working with multiple machine learning methods (support vector regression, random forest regression, and knn regression through scikit-learn) and want to know how to determine which method is most appropriate to pursue given incrementally increasing my training sample by 10,000 records.

My training data size is about 80k records, and my test dataset is approximately 30k records. I start with 10k records to train each of my models because the system I am working in has very limited cores (and runs on a VM on a different continent) so I can't simply start a training session and walk away for a few hours.

As I increase the training size, I expect that my training validation scores (RMSE, MAPE) will decrease on the full training dataset, and my testing validation scores will increase.

What are the metrics that I should capture about each run in order to make informed decisions about the complexity, speed, and long term viability of these machine learning models?

(This post does a great job of explaining the general steps associated with the *entire* data science workflow. However, I am particularly interested in exploring what would be considered the ideal set of metrics that I should capture after having trained, tested, and validated a machine learning model.)

Answer 1

For Modelling , it's probably best to go with Partial learning models. Explore DASK.

If you can, getting cloud support to run your models is also excellent idea. For examples Google colab.

For all models keep log of model accuracy and cross validation accuracy. It's good idea to keep these details in small table and build a line graph which shows accuracy details for each version.

For latency, use Pytest and build differential testing code so that after each iteration of testing it provide tesrit results.