3
$\begingroup$

I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model was built with Python and Keras. It takes in a CSV table of inputs and spits out the corresponding CSV table of predicted outputs.)

My naive "vision" is that I'd have the model reside on a single (master) machine, whereas the data would be equally distributed among several units (whatever unit means: nodes in a "cluster", CPUs, GPUs, ... ?) The model would be projected onto these units and the learned parameters would somehow all synchronize back to the master unit. Similarly, in the case of serving, the same model would be applied to the data that resides on the different units. Does this "vision" sound reasonable? (I have had some experience with parallel computing with MPI and I vaguely remember that's how things work.)

If I were to start from a blank slate, what architecture/infrastructure should I choose to deploy my model in a scalable fashion? Below are some of the confusingly many options I have read about. (I hardly master what each of these things do, so please forgive me if it looks like a laundry list of disparate technologies.)

As a pure ML guy (read: Python, Keras, pandas guy coding on a laptop), I'm out of my depth with all the infrastructure jargon that comes with the above links. It's therefore overwhelming to find a starting point, or some kind of "Hello World" example I could relate to. All I want is an architecture from which I can transpose my code and have it run in an efficient and scalable manner. Which one does the job? Despite all the hype around ML, there does not seem to be any comprehensive map or comparative review of all these solutions. I did find some comparisons, for example between Spark and SageMaker, or Spark and Dask, but given my illiteracy in these subjects, they only add to the confusion.

Improve this question
$\endgroup$
4
  • $\begingroup$ How about a courtesy explanation for the drive-by downvote? Thank you. $\endgroup$
    – Tfovid
    Sep 7, 2018 at 7:35
  • $\begingroup$ It sometimes happens, don't worry :) welcome to our community. $\endgroup$
    – Green Falcon
    Sep 7, 2018 at 8:28
  • 1
    $\begingroup$ Yeah this list is all over the place - notebooks, resource managers, DL libraries, storage, compute frameworks. It sounds like you want distributed DL frameworks only right? $\endgroup$
    – Sean Owen
    Sep 7, 2018 at 22:59
  • 1
    $\begingroup$ @SeanOwen It does not have to be deep learning. It could be something as simple as an SVM from Scikit-learn or a shallow net with Keras. With my current prototype, I feed in an input CSV and get an output CSV of predictions. Ultimately, it won't be a CSV off my own hard drive, but some massive production-scale database. I'm not an infrastructure guy, however, and for being the only 'data sciency' guy in my company, the onus is on me to bridge the gap between my current ML prototypes and a proposal of scalable infrastructure. $\endgroup$
    – Tfovid
    Sep 8, 2018 at 8:17

2 Answers 2

Reset to default
2
$\begingroup$

If you're looking for distributed training, you're probably looking for Apache Spark. It's not itself a training library (although it comes with Spark MLlib which implements most common algorithms in a distributed way), but Spark core is something other implementations often build on.

Pyspark is just the Python API for Spark. Mesos is a cluster manager, like YARN, that Spark can use. Hadoop is for our purposes here a storage system (HDFS) and YARN, so might have the data you're reading. Spark builds on that too.

Kubeflow is, AFAIK, a toolkit for running many relevant data science things on the Kubernetes resource manager. If you're using K8S this will help you get some common training and notebook tools running. I don't know how much it specifically assists distributing training.

Dask is a framework for distributing Python code but I haven't used it; I don't think it's a training library per se. Sagemaker is a hosted notebook and serving tool, mostly. (Plug: if you're interested in Sagemaker, but also using Spark, you'd probably like our Databricks platform even more.)

Many libraries have distributed implementations now. TensorFlow has its own distributed mode. Horovod from Uber also helps distributed TensorFlow in a more efficient way. There's TensorFlowOnSpark too. (Plug again, because this is near and dear to my heart: HorovodEstimator = Horovod + Spark)

MXnet, xgboost also have distributed implementations on Spark. There's BigDL and deeplearning4j, also distributed deep learning for Spark.

And there's more. You're probably going to run into Spark no matter what as its even more commonly used for ETL and other processing related to data science. Then pick your preferred platform. Then pick some distributed training tools to try on your problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. So just to have a better picture: If Spark is all I need, why would I bother with choosing a platform/cluster manager (e.g., Kubernetes, Mesos, SageMaker)? Is that for things like load balancing which Spark can't handle? $\endgroup$
    – Tfovid
    Sep 10, 2018 at 7:46
  • 1
    $\begingroup$ If you are running stand-alone Spark clusters, you can use its built-in resource manager. If you're running Spark on a shared cluster, you need something to manage resource allocation among Spark and another things. That's where Mesos, K8S, YARN come in. This is all about wanting to limit resources Spark uses, which may or may not be relevant to you. SageMaker, Zeppelin, Databricks, etc provide a notebook env on top (among other things), and that too is orthogonal to Spark or any particular modeling library. $\endgroup$
    – Sean Owen
    Sep 11, 2018 at 14:35
0
$\begingroup$

One of the things that affects your architecture is what you're parallelizing - e.g. are you taking an ML technique that does well on a GPU and splitting the work between GPUs? Or are you parallelizing cleaning of a giant dataset of images?

Spark is a big map-reduce system and I think most resembles what you described above. Sagemaker has pre-built algorithms with some optimizations, and those can set up all the libraries needed to run on a GPU for you, which can be time consuming.

When you say "deploy" a model you're talking about something potentially different - is it a web API (do you want to pay for GPU hosting?) is it running on a device (e.g. is the model tuned to fit well on a phone)?

Improve this answer
$\endgroup$
1
  • $\begingroup$ My bad---by deploy, I really meant "scale". In a nutshell, the situation is as follows: I have some ML code (Keras/scikit-learn) that ingests a CSV of data and spits out a CSV of predictions. All of this currently runs on my desktop and the CSVs aren't larger than a couple of gigabytes. I want to scale this to much larger data sets, eventually too large to sit or run on any single machine. I also want to opt for the most portable platform and with simple, bare-bone programming---i.e., without the overhead of learning a whole ecosystem. $\endgroup$
    – Tfovid
    Feb 7, 2019 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.