I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Define production here? what are you doing with it, what are your requirements? input, output form? $\endgroup$– Sean OwenMar 2, 2020 at 3:07
-
$\begingroup$ @SeanOwen I have a BERT model that takes CSV files as input (text) and it outputs text with some predictions (restored punctuation in this text). The model works pretty fast on GPU, but I run it as a python (PyTorch) script. For example, if I have 100 GPU how can I parallelize computation? I cannot use Spark with it because Spark doesn't include PyTorch. Is Docker my option here? So, here, production means parallelization of computations and running it in real-time. $\endgroup$– illuminatoMar 10, 2020 at 19:03
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You can just apply it with Spark. There is no reason you can't use Pytorch in a Spark job; just add it as a dependency when you submit the job. Spark's pandas UDFs can be pretty useful for scoring large models as they let you score in mini batches. See https://spark.apache.org/docs/3.0.0-preview/sql-pyspark-pandas-with-arrow.html#scalar-iterator
One complication is that you can use GPUs in Spark 2.x, but can't allocate GPUs as resources. So you may have multiple tasks on one GPU, and need to tune a little bit to reduce contention. Spark 3 however will have GPU resource allocation.
Hadoop isn't a thing that runs computations, unless you mean MapReduce, which is obsolete, or if you mean Spark, which is above.
Docker is also an option; just bottle up your scoring code and run on a cluster. You don't really get the same help with data movement and access that you would in Spark; it's all up to you. But sure that can work.
answered Mar 10, 2020 at 20:43