5
$\begingroup$

I'm applying ML for classification task in Python using sklearn/pandas. I'm going to try various things to get the best results, and I wonder how do I effectively store and analyze all the parameters and results of the classification? Parameters include:

  1. Number of training examples (which can be extended as I get more labeled data).
  2. Set of features.
  3. Classification algorithm.
  4. Algorithm hyperparameters.

Results include:

  1. Precision/recall for each of the classes.
  2. Overall precision/recall.
  3. Support for each class, etc.

Of course, I can manually copy the parameters and results to an Excel spreadsheet every time, but it's not an optimal solution. Are there any Python libraries (or modules of sklearn/pandas) which allow to easily store and display the parameters and results for later analysis? How do you solve this task?

$\endgroup$
5
$\begingroup$

Sacred is a python library developed by the IDSIA lab that "facilitates automated and reproducible experimental research". It is available through pip as sacred.

For a related discussion see reddit.

$\endgroup$
1
$\begingroup$

I ended up using Feature Forge, which is a python library that "provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, etc.), and particularly helpful if you use scikit-learn (although this can work if you have a different algorithm)".

I chose it because it seems to be a more specific tool for machine learning than Sacred, which appears to be a more generic tool. Feature Forge also provides a convenient framework for describing features in a scalable and reusable class-based way. It also has functions to store experimental results in a Mongo database.

Thanks to @Emre for pointing to reddit discussion, where featureforge was mentioned.

UPD:

I wrote a library based on Feature Forge to support the full machine learning experimentation pipeline:

  • classifier configuration management
  • feature description
  • training/testing classifiers
  • storing/analyzing results.

Feel free to contribute: https://github.com/goldan/machinery

$\endgroup$
  • $\begingroup$ Could you give an example experimentation setup that worked for you? $\endgroup$ – serv-inc Jun 29 '17 at 14:56
  • 1
    $\begingroup$ @serv-inc I created a library based on Feature Forge to support the full experimentation pipeline: classifier configuration management, feature description, training/testing classifiers and storing/analyzing results. Feel free to contribute: github.com/goldan/machinery $\endgroup$ – Dennis Golomazov Jul 5 '17 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.