2
$\begingroup$

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ on my local machine.

I am wondering what the best way would be to go about development of an application to run on the data that is in the cluster. Things on my mind for consideration are the following;

  1. IntelliJ is a nice IDE but I can't store the volume of data I want to look at on my local machine
  2. I can run spark-shell on the edge node but its not great for data exploration and development

How is it best to bridge this gap between data and a nice development environment?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

What you want to do is develop your code in IntelliJ, and then package your code and dependencies into an executable jar file using SBT or Maven.

When you have your jar stored locally, you can use spark-submit to transfer the jar to your cluster (along with some other parameters) for execution.

You might also want to take a sample of your data and store it locally so you can run spark locally and test/debug your code in IntelliJ. This can speed up development considerably, and having access to a debugger is a huge help.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.