In my opinion, I recommend to use the following approach to develop spark jobs for a big Data Context :
- First, develop your spark in local mode on your computer. Use a simple subset of data from your final dataset. This way allows you to work with your favorite IDE (jupiter notebook, PyCharm...) and small computing latences. This is really suitable for spark job development.
- Secondly, once your are done with dev in local mode, you can transfer your code on a cluster using SSH clients or whatever. You just have to change your spark context relative instructions and the input/output sources. Now test your code on a small dataset again and correct the few issues that could appear.
- Finally, once your code is cluster suitable, you can now use it on your whole Big Data like dataset.
Note : On a Hadoop based cluster, I recommend to store your data on HDFS ORC format. You can then smoothly structure it with Hive and easily read or write these tables with Spark.
Note : Concerning the SSH clients, I recommend you using mobaXterm (a ssh based client for windows) to easily access cluster and drag-and-drop your code. To get some subsets of data for local mode, you can use "Toad for Hadoop" client which gives you the possibility to run SQL-like queries directly on your hive tables and extract data in CSV or Excel format.
Note : Spark Standalone mode is not suitable for more that few machines. Prefer yarn or mesos clusters approaches for big datasets.