I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is: I have a logistic regression model for which I want to find the best regularization parameters (regParam and elasticNetParam). To do that, I use the CrossValidator which works and finds me a model better than all the other one I tried. The issue is that I do not know how to access the actual value of the parameters that were found by the cross validator. Below is the code I use to fit my cross validator:

from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.classification import LogisticRegression

lr_predictor = LogisticRegression(featuresCol='polyFeatures', labelCol='label', maxIter=10)
paramGrid = ParamGridBuilder() \
    .addGrid(lr_predictor.elasticNetParam, [0., 0.5, 1]) \
    .addGrid(lr_predictor.regParam, [0.1, 0.01]) \
crossval = CrossValidator(estimator=LogRegPipeline,
cvModel = crossval.fit(train_set)
bestModel = cvModel.bestModel
# How to get the best parameters fitted by cvModel 
  • $\begingroup$ Did you try bestModel.extractParamMap()? $\endgroup$ – Biggus Sep 6 '19 at 16:48

It's not the most elegant solution but you can use the following to at least zip together evaluation metrics and hyper parameters

zip(cvModel.avgMetrics, paramGrid)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.