# How to store and analyze classification results with Python?

I'm applying ML for classification task in Python using sklearn/pandas. I'm going to try various things to get the best results, and I wonder how do I effectively store and analyze all the parameters and results of the classification? Parameters include:

1. Number of training examples (which can be extended as I get more labeled data).
2. Set of features.
3. Classification algorithm.
4. Algorithm hyperparameters.

Results include:

1. Precision/recall for each of the classes.
2. Overall precision/recall.
3. Support for each class, etc.

Of course, I can manually copy the parameters and results to an Excel spreadsheet every time, but it's not an optimal solution. Are there any Python libraries (or modules of sklearn/pandas) which allow to easily store and display the parameters and results for later analysis? How do you solve this task?

• – Emre Mar 2 '16 at 22:35
• @Emre exactly what I was looking for. Please convert it to an answer, and I will happily accept it. – Dennis Golomazov Mar 2 '16 at 22:47
• Sacred is looking great, though even more valuable can be the featureforge app (see github.com/machinalis/featureforge), mentioned in reddit link, especially if you use scikit-learn. – Dennis Golomazov Mar 2 '16 at 22:50

Sacred is a python library developed by the IDSIA lab that "facilitates automated and reproducible experimental research". It is available through pip as sacred.

For a related discussion see reddit.

I ended up using Feature Forge, which is a python library that "provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, etc.), and particularly helpful if you use scikit-learn (although this can work if you have a different algorithm)".

I chose it because it seems to be a more specific tool for machine learning than Sacred, which appears to be a more generic tool. Feature Forge also provides a convenient framework for describing features in a scalable and reusable class-based way. It also has functions to store experimental results in a Mongo database.

Thanks to @Emre for pointing to reddit discussion, where featureforge was mentioned.

UPD:

I wrote a library based on Feature Forge to support the full machine learning experimentation pipeline:

• classifier configuration management
• feature description
• training/testing classifiers
• storing/analyzing results.

Feel free to contribute: https://github.com/goldan/machinery

• Could you give an example experimentation setup that worked for you? – serv-inc Jun 29 '17 at 14:56
• @serv-inc I created a library based on Feature Forge to support the full experimentation pipeline: classifier configuration management, feature description, training/testing classifiers and storing/analyzing results. Feel free to contribute: github.com/goldan/machinery – Dennis Golomazov Jul 5 '17 at 18:05