I have made a spark cluster containing 2 workers and 1 master. I followed the following link to set up the spark clusters. After successfully setting up the spark clusters I wanted to connect it to Zeppelin.

Initially I realized when code is run on zepplin it runs on an anonymous cluster generated within zepplin. But I finally configured it to run on my Spark Cluster.

I a simple python program on zepplin.

# create DataFrame from python list. It can infer schema for you.
df1 = spark.createDataFrame([(1, "andy", 20, "USA"), (2, "jeff", 23, "China"), (3, "james", 18, "USA")]).toDF("id", "name", "age", "country")

# create DataFrame from pandas dataframe
df2 = spark.createDataFrame(df1.toPandas())

This was not getting executed. The notebook kept it's state at pending. I believe there is a configuration problem.

I also realized that every simple code I entered wasn't executed. even print("Hello World"). I want to know the steps on how to exactly set up zepplin with the spark cluster.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.