Best practices to store Python machine learning models

Question

What are the best practices to save, store, and share machine learning models?

In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ~100Mo large. Also, joblib can save one model to multiple files unless you set compress=1 (https://stackoverflow.com/questions/33497314/sklearn-dumping-model-using-joblib-dumps-multiple-files-which-one-is-the-corre).

But then, if you want to control access rights to models, and be able to use models from different machines, what's the best way to store them?

I have a few choices:

• Store them as files, and then put them in a repository using Git LFS
• Store them in an SQL database as binary files:
  • For instance in Postgresql https://wiki.postgresql.org/wiki/BinaryFilesInDB
  • This is also the method recommended by the SQL Server team:
    • https://docs.microsoft.com/en-us/sql/advanced-analytics/tutorials/walkthrough-build-and-save-the-model
    • https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/step/3.html
    • https://blogs.technet.microsoft.com/dataplatforminsider/2016/10/17/sql-server-as-a-machine-learning-model-management-system
• HDFS

Answer 1

You may have a look at nexus or dvc or datmo.

There was recently a presentation at at meetup in berlin, zalandos AI data engineering meetup.

Answer 2

I would like to suggest 2 more approaches.

1. Store them in document storage (eg. mongoDB) - this method is recommended when your model files are less then 16Mb (or the joblib shards are), then you can store model as binary data. in addition, some ML libraries support model export and import in json (eg. LightGBM), which makes it a perfect candidate for storage in document storage. Advantages: easy tracking of model generation and easy access, Disadvantages: things will get messy if model object is too large.

2. Store your model on object storage (eg. Amazon S3) - this method is good if your models are very large, in this case you get unlimited storage and fairly easy API, you pay more, that is for sure. Advantages: Unlimited space and ability to store arbitrary file formats. Disadvantages: cost, and the fact that to do it right you'll need to develop your own tracking system.

good luck!

Answer 3

I faced this problem (and still face it today) for many years. I really thing that, if you don't provide detailed requirements, you can't expect a serious answer. I explain myself with examples of my work:

• I regularly try multiple variations of the same model to find what parameters work best. It takes several day to train one single model which produces some output that is later used for evaluation. To do so, I make a simple NumPy dump of the model since it is easy to share it between servers, or colleagues. You should avoid pickle since it stores much more (instances of class, libraries...) than just the parameters learned by your model. Importing the model on another machine might not work if the python environment slightly differs.

• When pushing a model in production, I need 1) a version of the model that I can load fast in case of a server breakdown (typically a binary format, storing only what is necessary such as weights of a neural network) and 2) a way to keep the model in-RAM to quickly deal with the API requests.

For two different purposes, I need three different formats. Then, more generally speaking, the choice of the format depends on the tools you use. For example, if you work with TensorFlow, you might be interested in their TensorFlow Serving system

Answer 4

I have faced this issue for many years, as a data scientist or an ML engg. we have to create a ton of models before coming to a conclusion. Effectively storing all the model's profiles, parameters, and features used is a pain specially if you have multiple notebooks.

modellogger is a python package that can help you to organise the stuffs and create a full blown summary with dynamic stats on the call of an function.

pip install modellogger

follow the 3 steps easy documentation and voila you have it.

Answer 5

I had also run into this problem several times, so I've created an open source modelstore Python library which seeks to tackle the problem of simplifying the best practices around versioning, storing, and downloading models.

As others have pointed out, the best practices around this area are still forming. The whole area is fairly straightforward if you are saving one model, but starts becoming complicated as you need to store many models or newly trained versions of existing models. I've seen teams save models into shared drives, Cloud Storage/s3 buckets or git repos, manually or using ad hoc scripts.

To unify this, there are options like MLFlow's artifact storage which is great if you can set up and maintain a tracking server. Or, as others have mentioned, there are tools like DVC.

The modelstore library unifies the versioning and saving of an ML model into a single upload() command, and also provides a download() function to get that model back from storage. Here is (broadly) what it looks like:

from modelstore import ModelStore

modelstore = ModelStore.from_aws_s3(os.environ["AWS_BUCKET_NAME"])

model = train() # Replace with your code

# Here's an sklearn example - the library currently supports 9 different ML frameworks
modelstore.sklearn.upload("my-model", model)

The upload() command will create a tar archive containing your model and some meta-data about it, and upload it to a specific path in your storage.