I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (val model), then the pipeline is saved to a directory. The next line is
val sameModel = PipelineModel.load("/tmp/spark-logistic-regression-model")
I don't see how/where sameModel is being used. I see model being used again on test data. I would've expected the example to use sameModel to show how to invoke the loaded pipeline back into the process. Does sameModel automatically update model?
Any insight would be appreciated as I am interested in saving a pipeline following a fit then load it at a later point, but am having issues with:
- Finding good examples saving & loading Spark ML pipelines, then invoking said loaded pipeline
- Not fully understanding the examples I see online
Any assistance in clarifying what is going on and how to move forward would be greatly appreciated. Thanks!