# Machine Learning in Spark

I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get some snippet in python to find the probability of labels.

• Could you be less vague and provide some actual context? What model are you using? Which function from which library? What are you trying to predict? – Alex R. Jun 21 '16 at 23:31

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then model.transform(test) contains a node of probability, for more details please check the below code, going to show you the probability node and others useful nodes also for iris dataset.

Partition dataset randomly into Training and Test sets. Set seed for reproducibility

(trainingData, testData) = irisdf.randomSplit([0.7, 0.3], seed = 100)

trainingData.cache()
testData.cache()

print trainingData.count()
print testData.count()


Output:

103
47


Next, we will use the VectorAssembler() to merge our feature columns into a single vector column, which we will be passing into our Naive Bayes model. Again, we will not transform the dataset just yet as we will be passing the VectorAssembler into our ML Pipeline.

from pyspark.ml.feature import VectorAssembler
vecAssembler = VectorAssembler(inputCols=["SepalLength", "SepalWidth", "PetalLength", "PetalWidth"], outputCol="features")


For iris dataset, it has three classes namely setosa, versicolor and virginica. So let's create a Multiclass Naive Bayes Classifier using pysaprk library ml.

from pyspark.ml.classification import NaiveBayes
from pyspark.ml import Pipeline

# Train a NaiveBayes model
nb = NaiveBayes(smoothing=1.0, modelType="multinomial")

# Chain labelIndexer, vecAssembler and NBmodel in a pipeline
pipeline = Pipeline(stages=[labelIndexer, vecAssembler, nb])

# Run stages in pipeline and train model
model = pipeline.fit(trainingData)


Analyse the created mode model, from which we can make predictions.

predictions = model.transform(testData)
# Display what results we can view
predictions.printSchema()


Output

root
|-- SepalLength: double (nullable = true)
|-- SepalWidth: double (nullable = true)
|-- PetalLength: double (nullable = true)
|-- PetalWidth: double (nullable = true)
|-- Species: string (nullable = true)
|-- label: double (nullable = true)
|-- features: vector (nullable = true)
|-- rawPrediction: vector (nullable = true)
|-- probability: vector (nullable = true)
|-- prediction: double (nullable = true)


You can also select a particular node to view for some dataset as:

# DISPLAY Selected nodes only
display(predictions.select("label", "prediction", "probability"))


Above will show you in tabular formate.

Reference:

• In the above probability node, actual label value is 1.0 but the predicted label is 0.0 because the probability of 0.0 label is highest, its clearly visible from the probability node. anyway its wrong prediction. – krishna Prasad Jun 22 '16 at 11:37
• hi @krishna prasad when i give model.transform(train) it says naivebayesmodel has no attribute named transform. – Vignesh Mohan Jun 23 '16 at 11:02
• @VigneshMohan Please check it now, I have changed the model now its proper way of doing ML using pipeline. Hope this will work for you. – krishna Prasad Jun 24 '16 at 2:09
• @VigneshMohan Did you able to replicate it? – krishna Prasad Jun 27 '16 at 11:41