Can someone provide a summary of the real world deployment experience of MLflow? We have a few ML models (e.g., LightGBM, tensorflow v2, etc.) and want to avoid framework like SageMaker (due to customer requirement). So we are looking into various ways of hosting ML models for inferencing. Latency is one key performance metric that is very important to us. MLflow looks like a good choice for us.

It would be greatly appreciated if the users of MLflow can share their experience.


1 Answer 1


I am currently using MLflow together with Databricks at my company, and up to now, I would say it is quite ok for managing models versions, tracking models metrics and building the models workflow from a dev/experimenting/training status to production. I am not sure if it can be considered a standalone tool for that, meaning that you should combine it with something like Azure Databricks (what I currently use). It's nice the way you can customize the tags, metrics or artifacts that you might want to keep logged after each run or model training, as follows: enter image description here

where you can also visualize several run experiments as:

enter image description here (these models are for the shake of the example, so metric values are not optimized) and storing the generated models files (e.g. a pickle file) in the corresponding location: enter image description here

Once it is generated, you can register your best model via the databricks UI (in case you finally use it) or via mlflow code as follows (here I register the last one, you should take the one with the best prefered metric which mlflow also gives you directly as an option): enter image description here

so as to be loaded later on and make predictions with your model. Once you your are happy with some model, you can transition it: enter image description here

Another nice feature I like is that it offers you custom modules for the most used frameworks like scikitlearn, xgboost, keras, etc

and build a container image, deploy it... but this is another longer topic.

Of course, you can also use MLflow with another third party or even locally.

About the performance point, do you mean the inference itself or some other source of latency? You can make inferences with your model via SPARK UDFs (PySPARK in my case) info, in case you need several predictions as fast as possible or simply for batch inference.

  • 1
    $\begingroup$ Thanks for the comment. Regarding the performance point, I am referring to the inference latency. For example, if you use PySPARK to query the model in MLflow, what is the elapsed time that one can expect before the response of the query is received? $\endgroup$ Commented Dec 2, 2020 at 0:25
  • 1
    $\begingroup$ the latency you ask for actually depends on many other (engineering) factors, like which cluster type you have, and in general which infrastructure you play with, but, if we only talk about the chances you have with MLflow, you should take a look at its API: mlflow.org/docs/latest/python_api/… + mlflow.org/docs/latest/python_api/… ... $\endgroup$
    – German C M
    Commented Dec 2, 2020 at 11:44
  • 1
    $\begingroup$ Typical recommendation is to use Spark for batch inference, or inference on the streaming data (microbatch) that allows to get ~1 seconds response time. If you need to go to the milliseconds area, MLflow allows to deploy to Azure ML, SageMaker, pack as Docker, or on Databricks use the model serving on dedicated cluster. $\endgroup$
    – Alex Ott
    Commented Dec 11, 2020 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.