Pyspark code is not performant enough when compared to pure python alternative

Question

I transformed the existing code which was in python pasted below was in pyspark.

Python code:

import json
import csv


def main():
    # create a simple JSON array
    with open('paytm_tweets_data_1495614657.json') as str:

        tweetsList = []
        # change the JSON string into a JSON object
        jsonObject = json.load(str)

        #print(jsonObject)

        # # print the keys and values
        for i in range(len(jsonObject)):
            tweetsList.insert(i,jsonObject[i]["text"])

        #print(tweetsList)
    displaySentiment(tweetsList)



def displaySentiment(tweetsList):
    aDict = {}

    from sentiment import sentiment_score

    for i in range(len(tweetsList)):
        aDict[tweetsList[i]] = sentiment_score(tweetsList[i])
    print (aDict)


    with open('PaytmtweetSentiment.csv', 'w') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames = ["Tweets", "Sentiment Value"])
        writer.writeheader()
        writer = csv.writer(csv_file)
        for key, value in aDict.items():
            writer.writerow([key, value])


if __name__ == '__main__':
    main()

Converted Pyspark Code:

import json
import csv
import os
from pyspark import SparkContext, SparkConf
from pyspark.python.pyspark.shell import spark

os.environ['PYSPARK_PYTHON'] = "/usr/local/bin/python3"


def main():
    path = "/Users/i322865/DeepInsights/bitbucket-code/ai-engine/twitter-sentiment-analysis/flipkart_tweets_data_1495601666.json"
    peopleDF = spark.read.json(path).rdd
    df = peopleDF.map(lambda row: row['text'])
    print(df.collect())
    displaySentiment(df.collect())



def displaySentiment(tweetsList):
    from sentiment import sentiment_score

    aDict = sentiment_score(tweetsList)

    #
    with open('paytmtweetSentiment.csv', 'w') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames = ["Tweets", "Sentiment Value"])
        writer.writeheader()
        writer = csv.writer(csv_file)
        for i in range(len(tweetsList)):
            writer.writerow([tweetsList[i], aDict[i]])
            print([tweetsList[i], aDict[i]])


if __name__ == '__main__':
    conf = SparkConf().setAppName("Test").setMaster("local")
    sc = SparkContext.getOrCreate(conf=conf)
    main()

I ran both programs but didn't see any significant performance improvement. What am I missing? Please could you shed some thoughts?

Answer 1

You have instantiated a sparkContext object with "local" mode configuration. It means you have allocated ressources for a single multi-core Java Virtual Machine on your computer. In this configuration, you can't reach better performs than with python. Because :

• With local mode, you have less ressources (cpu, memory...) with Spark than with python (you can't have a JVM with more ressources than your computer).
• PySpark code is converted into Scala code before execution.

Spark benefits arise when you use Spark on multiple nodes. In this configuration, spark master is yarn, mesos or standalone. In that case, a spark job would be separated in multiple tasks and each node would be dedicated to different tasks.

For example, if you have 3 nodes and 6 tasks, each node can handle 2 tasks. If you have 6 nodes and 6 tasks, each node will handle a task. With simple python, think that you have only 1 node for "6 tasks". So, for large tasks (big enough datasets), Spark will results in better latencies than simple python.