I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum message size:

    scala> val mod = pipeline.fit(df)
    mod: org.apache.spark.ml.PipelineModel = pipeline_936bcade4716
    scala> mod.write.overwrite().save(modelPath.concat("model"))
    18/01/08 10:00:32 WARN TaskSetManager: Stage 8 contains a task of very large size 
    (755610 KB). The maximum recommended task size is 100 KB.
    org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 
    2606:0 was 777523713 bytes, which exceeds max allowed: spark.rpc.message.maxSize 
    (134217728 bytes). Consider increasing spark.rpc.message.maxSize 
    or using broadcast variables for large values.

Making the following assumptions about the problem area:

  1. It's not possible to decrease the size of the model AND
  2. It's not possible to increase the maximum message size to a point where the pipeline would fit in a single message.

Are there any methods that would allow me to save the pipeline successfully to HDFS?


1 Answer 1


You can do the following:

A Pipeline can be made of other pipelines. Isn't that great? A Pipeline inherit from the Estimator class and by definition, a PipelineStage can be either an Estimator or a Transformer.

This way, you can build smaller pipelines, save them separately and on the other software/class, join them again as a single one and call transform on the DataFrame.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.