I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin notebook online. So I write my Spark code in a browser window in the moment, which I find not very handy.
I wonder what options are avalable to code on my local computer and "push" tasks to the remote Spark/Hadoop cluster?
So instead of coding in a browser window, I would be able to use (for instance) RStudio, write SparkR code, and execute the code on the remote cluster.
Since I'm rather new to this architecture, I'm thankful for any hints.
Note: This question is not about retrieving data from the Hadoop cluster. I already have a OCDB connection, so that I can can make SQL like queries.
spark_submit$\endgroup$