0
$\begingroup$

I have made a spark cluster containing 2 workers and 1 master. I followed the following link to set up the spark clusters. After successfully setting up the spark clusters I wanted to connect it to Zeppelin.

Initially I realized when code is run on zepplin it runs on an anonymous cluster generated within zepplin. But I finally configured it to run on my Spark Cluster.

I a simple python program on zepplin.

# create DataFrame from python list. It can infer schema for you.
df1 = spark.createDataFrame([(1, "andy", 20, "USA"), (2, "jeff", 23, "China"), (3, "james", 18, "USA")]).toDF("id", "name", "age", "country")
df1.printSchema
df1.show()

# create DataFrame from pandas dataframe
df2 = spark.createDataFrame(df1.toPandas())
df2.printSchema
df2.show()

This was not getting executed. The notebook kept it's state at pending. I believe there is a configuration problem.

I also realized that every simple code I entered wasn't executed. even print("Hello World"). I want to know the steps on how to exactly set up zepplin with the spark cluster.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.