I have made a spark cluster containing 2 workers and 1 master. I followed the following link to set up the spark clusters. After successfully setting up the spark clusters I wanted to connect it to Zeppelin.
Initially I realized when code is run on zepplin it runs on an anonymous cluster generated within zepplin. But I finally configured it to run on my Spark Cluster.
I a simple python program on zepplin.
# create DataFrame from python list. It can infer schema for you. df1 = spark.createDataFrame([(1, "andy", 20, "USA"), (2, "jeff", 23, "China"), (3, "james", 18, "USA")]).toDF("id", "name", "age", "country") df1.printSchema df1.show() # create DataFrame from pandas dataframe df2 = spark.createDataFrame(df1.toPandas()) df2.printSchema df2.show()
This was not getting executed. The notebook kept it's state at pending. I believe there is a configuration problem.
I also realized that every simple code I entered wasn't executed. even
print("Hello World"). I want to know the steps on how to exactly set up zepplin with the spark cluster.