I am working with multiple machine learning methods (support vector regression, random forest regression, and knn regression through scikit-learn) and want to know how to determine which method is most appropriate to pursue given incrementally increasing my training sample by 10,000 records.

My training data size is about 80k records, and my test dataset is approximately 30k records. I start with 10k records to train each of my models because the system I am working in has very limited cores (and runs on a VM on a different continent) so I can't simply start a training session and walk away for a few hours.

As I increase the training size, I expect that my training validation scores (RMSE, MAPE) will decrease on the full training dataset, and my testing validation scores will increase.

What are the metrics that I should capture about each run in order to make informed decisions about the complexity, speed, and long term viability of these machine learning models?

(This post does a great job of explaining the general steps associated with the *entire* data science workflow. However, I am particularly interested in exploring what would be considered the ideal set of metrics that I should capture after having trained, tested, and validated a machine learning model.)

  • $\begingroup$ "[...] I can't simply start a training session and walk away for a few hours". People do it all the time, with the help of, for example, linux, ssh and screen. $\endgroup$
    – Valentas
    Mar 26 at 5:31

1 Answer 1


For Modelling , it's probably best to go with Partial learning models. Explore DASK.

If you can, getting cloud support to run your models is also excellent idea. For examples Google colab.

For all models keep log of model accuracy and cross validation accuracy. It's good idea to keep these details in small table and build a line graph which shows accuracy details for each version.

For latency, use Pytest and build differential testing code so that after each iteration of testing it provide tesrit results.

  • $\begingroup$ I’m adding a link to Dask. Could you provide an example or blurb on what you mean by “differential testing code?” $\endgroup$
    – shadow_dev
    Aug 30, 2019 at 21:38
  • $\begingroup$ Hi @riyaj - I've searched for "partial learning models" to little benefit... could you please update your answer with more details and references? $\endgroup$
    – shadow_dev
    Sep 3, 2019 at 4:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.