SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?

Question

I am using LogisticRegressionWithLBFGS to train a multi-class classifier.

Is there a way to get the probability of all classes (not only the top candidate class) when I test the model on new unseen samples?

P.S. I am not necessarily obliged to use the LBFGS classifier but I would like to use the logistic regression in my problem. So if there is a solution by using another LR classifier type, I would go for it.

Answer 1

I am working on Random Forest Classifier and this classifier has probability attribute in prediction i.e if you get the summary of predictions = model.transform(testData) as print(predictions) in PySpark you will get the probability of each labels, You can check the below code and output of the code:

from pyspark.sql import DataFrame
from pyspark import SparkContext, SQLContext
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.feature import StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a Random Forest model.
rf = RandomForestClassifier(labelCol="label", featuresCol="features", numTrees=12,  maxDepth=10)

# Chain RF in a Pipeline
pipeline = Pipeline(stages=[rf])

# Train model.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

Now your work start from here. try printing predictions and values of predictions

print(predictions)

Output:

DataFrame[label: double, features: vector, indexed: double, rawPrediction: vector, probability: vector, prediction: double]

So, in the DataFrame you have the probability which is the probability of each indexedLabel, Further I have checked it as:

print predictions.show(3)

Output:

+-----+--------------------+-------+--------------------+--------------------+----------+
|label|            features|indexed|       rawPrediction|         probability|prediction|
+-----+--------------------+-------+--------------------+--------------------+----------+
|  5.0|(2000,[141,260,50...|    0.0|[34.8672584923246...|[0.69734516984649...|       0.0|
|  5.0|(2000,[109,126,18...|    0.0|[34.6231572522266...|[0.69246314504453...|       0.0|
|  5.0|(2000,[185,306,34...|    0.0|[34.5016453103805...|[0.69003290620761...|       0.0|
+-----+--------------------+-------+--------------------+--------------------+----------+
only showing top 3 rows

Only for probability column:

print predictions.select('probability').take(2)

Output:

[Row(probability=DenseVector([0.6973, 0.1889, 0.0532, 0.0448, 0.0157])), Row(probability=DenseVector([0.6925, 0.1825, 0.0579, 0.0497, 0.0174]))]

In my case I have 5 indexedLabels and so the probability vector length is 5, Hope this will help you to get the probability of each labels in your problem.

P.S: You will probably get the probability in Decision Tree, Logistic regression. Just try to get the summary of model.transform(testData).

Answer 2

To get all probabilities instead of all classes instead of just the labeled class, there is no explicit method till now (Spark 2.0) in Spark MLlib or ML. But you can extend the Logistic Regression class from the MLlib source code to get those probabilities.

A sample code snippet can be found in this answer.