Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data.

Ok, lets say I have one dataframe and one regressor, here what I do:

  1. Split the data.
  2. Apply the regressor to the training data.
  3. Apply the model to testing data to get predictions.
  4. Evaluate the prediction using an evaluator (e.g., binary classifier).

Now, lets say I have 3 steps and the last one is the regressors. I can not do the above because the first two also need to be applied to the testing data as well, how to deal with this? How my steps will look like?

  • $\begingroup$ What do you mean, the last one is the regressors? Which parts of the "above"? maybe include some sample code? $\endgroup$ – fractalnature Aug 13 '19 at 20:04

A pyspark.ml.PipelineModel is the result of calling the .fit() method of a pyspark.ml.Pipeline. This estimator is a sequence of Transformers and/or Estimators, all of which have a .transform() method. When you call the .transform() method of the pyspark.ml.PipelineModel-- say, when you want to make predictions using the trained model-- the .transform() method of each of the objects in the PipelineModel are called in sequence.

In other words, the transformations for each step are applied by calling the .transform() method of the fitted pipeline- the API does this for you, or more accurately, gives you concise tools to do this as you need to.

What you're talking about is slightly outside the scope of this type of object, however. Train/test splitting occurs outside the Pipeline, and importantly so, for methodological reasons.

A PipelineModel couples data preparation steps with an estimator, but not training operations. Splitting the data, for example, is something you'd have to do outside the Pipeline.

The workflow might go:

Split the data into training and test datasets.

Define your data preparation steps and regression estimator, and combine these into a Pipeline.

Train the entire pipeline on your training data:

fit_pipeline = Pipeline.fit(train)

Make predictions on the training set:

train_preds = fit_pipeline.transform(train)

Use something like a pyspark.ml.evaluation.RegressionEvaluator to evaluate the performance on the training dataset:

evaluator = RegressionEvaluator()

Then, you can make predictions on the test dataset:

test_preds = fit_pipeline.transform(test)

And evaluate them:


But you have this completed model and data preparation object that can be used to make predictions on similarly structured data now, called fit_pipeline, which is an instance of pyspark.ml.PipelineModel.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.