4
$\begingroup$

I am wondering what is the best practice other devs are using for their python spark jobs. I am building a dev environment in which I am looking to write code in PyCharm with SparkContext pointing to a standalone cluster and being able to run my code from PyCharm or at least from the command line onto the cluster. I am using Windows 8.

Any success stories with such set up? what is the recommended setup/practice?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In my opinion, I recommend to use the following approach to develop spark jobs for a big Data Context :

  • First, develop your spark in local mode on your computer. Use a simple subset of data from your final dataset. This way allows you to work with your favorite IDE (jupiter notebook, PyCharm...) and small computing latences. This is really suitable for spark job development.
  • Secondly, once your are done with dev in local mode, you can transfer your code on a cluster using SSH clients or whatever. You just have to change your spark context relative instructions and the input/output sources. Now test your code on a small dataset again and correct the few issues that could appear.
  • Finally, once your code is cluster suitable, you can now use it on your whole Big Data like dataset.

Note : On a Hadoop based cluster, I recommend to store your data on HDFS ORC format. You can then smoothly structure it with Hive and easily read or write these tables with Spark.

Note : Concerning the SSH clients, I recommend you using mobaXterm (a ssh based client for windows) to easily access cluster and drag-and-drop your code. To get some subsets of data for local mode, you can use "Toad for Hadoop" client which gives you the possibility to run SQL-like queries directly on your hive tables and extract data in CSV or Excel format.

Note : Spark Standalone mode is not suitable for more that few machines. Prefer yarn or mesos clusters approaches for big datasets.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I like this answer. I would also just add that the local copy the libraries and version should match with what's on your server. I prefer working with docker images of my distribution. That way I don't have to manually check versions. $\endgroup$
    – Tophat
    Jan 12, 2018 at 13:12
  • $\begingroup$ Could you point out a tutorial which may describe the setup more in detail mainly using Spark Structured Streaming and PySpark locally?? $\endgroup$
    – lony
    Feb 12, 2018 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.