0
$\begingroup$

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (val model), then the pipeline is saved to a directory. The next line is

val sameModel = PipelineModel.load("/tmp/spark-logistic-regression-model")

I don't see how/where sameModel is being used. I see model being used again on test data. I would've expected the example to use sameModel to show how to invoke the loaded pipeline back into the process. Does sameModel automatically update model?

Any insight would be appreciated as I am interested in saving a pipeline following a fit then load it at a later point, but am having issues with:

  1. Finding good examples saving & loading Spark ML pipelines, then invoking said loaded pipeline
  2. Not fully understanding the examples I see online

Any assistance in clarifying what is going on and how to move forward would be greatly appreciated. Thanks!

$\endgroup$
1
$\begingroup$

It appears that "sameModel" is just an example of loading the model back without any influence on "model" from the docs.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.