2
$\begingroup$

I have 500+ models predicting various things and a massive database of over 400m+ individuals and about 5,000 possible independent variables.

Currently, my scoring process takes about 5 days, and operates by chunking up the 400m+ records into 100k-person pieces and spinning up n-number of threads, each with a particular subset of the 500+ models, and running this way until all records are scored for all models. Each thread is a Python process which submits R code (i.e. loads an R .rds model and associated dataset transformation logic).

This process takes way too long, is severely error-prone (more of an indicator of the tangled web of code it's become), is expensive (massive cloud instance required), and only allows for models to be built in R (I want to basically be agnostic of the language from which the model is coming, but mainly I want to enable Python and R – that is a non-negotiable requirement).

Does anyone with experience in a similar problem domain have any advice re: how this process could be re-architected to 1) run more efficiently (from a $ PoV) and 2) enable both Python and R models.

$\endgroup$
2
  • $\begingroup$ By scoring do you mean evaluating or predicting? Because for evaluation purposes you could certainly run your models on a smaller sample. In general if you must run all the models on all the instances there's probably little to optimize at the technical level, it's just a trade-off between speed and price (dividing into many smaller chunks will go faster but be more expensive). $\endgroup$
    – Erwan
    Jan 21, 2020 at 21:27
  • $\begingroup$ By scoring I mean predicting – actually using a trained model in whatever business/application context for which it was designed. $\endgroup$
    – blacksite
    Jan 22, 2020 at 13:34

3 Answers 3

0
$\begingroup$

Since in the comments you mentioned making predictions based on trained models, the number data examples is not a factor. Training data can be ignored; You only need to use the trained model architecture and weights.

You probably want to use an existing distributed machine learning framework such as Spark or H20. That framework will hand distributing the predictions across the cluster and aggregating the results.

$\endgroup$
0
$\begingroup$

You can use Python Frameworks for Parallel and Distributed Machine Learning Tasks

for examle:Elephas is an extension of Keras, which allows you to run distributed deep learning models at scale with Spark. Elephas intends to keep the simplicity and high usability of Keras, thereby allowing for fast prototyping of distributed models, which can be run on massive data sets. Installation:

pip install elephas

for more education go to : 10 Python Frameworks for Parallel and Distributed Machine Learning Tasks

$\endgroup$
0
$\begingroup$

Spark has a library to run distributed machine learning tasks (Spark MLlib). You can use a Spark frontend for R (SparkR) or Python (PySpark) to train your models. Take a look: https://spark.apache.org/docs/latest/sparkr.html#machine-learning

Also, you can try to find a ML lib that provides a native API to train a model in Spark. Here's a quick start to train a classification or regression model using the Catboost lib (boosting) with PySpark: https://catboost.ai/en/docs/concepts/spark-quickstart-python

Alternatively, you can combine native R algorithms and SparkR User-Defined Functions (UDF) to distribute your tasks: "100x Faster Bridge between Apache Spark and R with User-Defined Functions on Databricks" (Databricks).

Here's an example how to perform grid search in a distributed way using a Sklearn standardlized algorithm and PySpark (you can adapt it for your needs with R and SparkR!): "Leveraging Machine Learning Tasks with PySpark Pandas UDF" (Neoway).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.