I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ on my local machine.
I am wondering what the best way would be to go about development of an application to run on the data that is in the cluster. Things on my mind for consideration are the following;
- IntelliJ is a nice IDE but I can't store the volume of data I want to look at on my local machine
- I can run spark-shell on the edge node but its not great for data exploration and development
How is it best to bridge this gap between data and a nice development environment?