How does real world machine learning production systems run?

Question

Dear Machine Learning/AI Community,

I am just a budding and aspiring Machine Learner who has worked on open online data sets and some POC's built locally for my project. I have built some models and converted into pickle objects in order to avoid re-training.

And this question always puzzles me. How does a real production system work for ML algorithms?

Say, I have trained my ML algorithm with some millions of data and I want to move it to production system or host it on a server. In real world, do they convert into pickle objects? If so, it would be huge pickled file, isn't. The ones I trained locally and converted for 50000 rows data itself took 300 Mb space on disk for that pickled object. I don't think so this is right approach.

So how does it work in order to avoid my ML algorithm to re-train and start predicting on incoming data? And how do we actually make ML algorithm as a continuous online learner. For example, I built a image classifier, and start predicting the incoming images. But I want to again train algorithm by adding the incoming online images to my previously trained data sets. May be not for every data, but daily once I want to combine all received data for that day and re-train with newly 100 images which my previously trained classifier predicted with actual value. And this approach shouldn't effect my previously trained algorithm to stop predicting incoming data as this re-training may take time based on computational resources and data.

I have Googled and read many articles, but couldn't find or understand to my above question. And this is puzzling me every day. Do manual intervention is needed for production systems as well? or any automated approach is there for it?

Any leads or answers to above questions would be highly helpful and appreciated. Please let me know if my questions doesn't make sense or not understandable.

This is not a project centric I am looking for. Just a generic case of real world production ML systems example.

Thank you in advance!

Answer 1

There are many things to consider to have a model in production. The main ones your are asking about are:

• Functionality
• Architecture

Functionality

For your model to be used in production from a web server, you can host an API which exposes your model.

For example, you have a Flask Python server running, where you map an endpoint (e.g. GET http://<your_host>/prediction/image.jpg) to the predict() function of your model.

Then you mentioned making it a continuous online-learner. Most classifiers will improve with more data if that data is annotated (i.e. labeled), but for that, you need to manually annotate them and re-feed them to your system and retrain your model. If you could automatically confidently label new data, you wouldn't need to improve your system. So, I would say, some manual labor would be required (labeling), but the rest can be automated. You can add more end-points to your web server, where you can upload more training data, and the system re-trains your model, takes care of versioning and re-loads the latest trained model.

Architecture

Storage

You mention pickle files and you are afraid that they are too large on disk. However, nowadays, with cloud solutions, this is often not a problem.

You can use Blob-Storage solutions and prices are often very low (e.g. https://azure.microsoft.com/en-us/services/storage/blobs/ will cost $0.002$euros/GB/month).

You can, of course, keep many pickles there, for versioning (recommended). However, if you want to minimize costs, you could only store the latest model.

Further, if your API is used often, you don't want to keep reloading your model every time. It would be better to have it always available in RAM. It is not expensive, again, to host a server with a lot of RAM in the cloud.

Layout

An architecture layout you can have is:

+----------------+          +--------------+
|                |          |              |
|  ADMIN SERVER  | -------> | BLOB STORAGE |
|                |          |              |
+----------------+          +--------------+
        |                           ^
        |                           |
        |               +-----------+-----------+
        |               |                       |
        |      +------------------+  +----------------+
        |      |                  |  |                |
        |      |  PREDICT SERVER  |  | PREDICT SERVER |
        |      |                  |  |                |
        |      +------------------+  +----------------+
        |                    ^          ^
        |                    |          |
        |                +------------------+
        |                | |              | |
        +--------------> | |     QUEUE    | |
                         | |              | |
                         +------------------+

Here, the ADMIN SERVER takes care of all the functionalities of re-training the model and uploading new models to the storage and publishing jobs to the queue for the PREDICT SERVERS to fetch the latest models from BLOB STORAGE.

The BLOB STORAGE holds the models.

The PREDICT SERVERs expose your predict() funtion, so your model is accessible to other systems. Here, the models are stored in RAM for faster predictions. Depending on the usage of your model, you might want to have $\geq1$ server for predictions. Since you model is persisted on BLOB STORAGE and not on your local hard-disk, this is possible, they can all fetch the latest model.

The QUEUE is how the ADMIN SERVER can communicate with all PREDICT SERVERs.

Answer 2

Yes, it is typical to have some persistent representation of the model that is uploaded, and yes, it is typically very big as a file/files. Using pickle is one way to do it, commonly used with scikit-learn, for example. Deep learning frameworks typically have their own formats, but nothing stops you from using pickle with them as well, except that it is more complicated and less efficient as an approach.

I'm not sure I understand the second part of the question, but if you want to modify your model online, nothing stops you from creating a new pickle. It is advisable that you run a new training as a separate batch process in the background, to avoid blocking your site or web service, especially if your web application server is single-threaded. Furthermore, you need to be wary of utilisation of server resources, so you would be better off running batch updates like that in low-traffic periods. This could be on over the weekend, or if your user base is predominantly regional, at night or on public holidays.