11
$\begingroup$

I'm using Spark (1.5.1) from an IPython notebook on a macbook pro. After installing Spark and Anaconda, I start IPython from a terminal by executing: IPYTHON_OPTS="notebook" pyspark. This opens a webpage listing all my IPython notebooks. I can select one of them, opening it in a second webpage. SparkContext (sc) is available already, and my first command in the notebook is help(sc), which runs fine. The problem I'm having is that I am getting a Java heap space error that I don't know how to address. How do I view my current Java heap setting, and how do I increase it within the context of my setup. The error message I'm getting follows:

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 19 in stage 247.0 failed 1 times, most recent failure: Lost task 19.0 in stage 247.0 (TID 953, localhost): java.lang.OutOfMemoryError: Java heap space
Improve this question
$\endgroup$
2
  • $\begingroup$ when are you having the error? trying to do what? $\endgroup$
    – eliasah
    Oct 23, 2015 at 6:30
  • $\begingroup$ Create an RDD of LabeledPoint. It is not particularly huge, 100K observations x2K feature vector. $\endgroup$
    – Kai
    Oct 23, 2015 at 17:06

3 Answers 3

Reset to default
19
$\begingroup$

You can manage Spark memory limits programmatically (by the API).

As SparkContext is already available in your Notebook:

sc._conf.get('spark.driver.memory')

You can set as well, but you have to shutdown the existing SparkContext first:

conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
        .set('spark.executor.memory', '4G')
        .set('spark.driver.memory', '45G')
        .set('spark.driver.maxResultSize', '10G'))
sc = SparkContext(conf=conf)

If your workload is the same for all analysis, then editing spark-defaults.conf as cited above is the way to go.

Improve this answer
$\endgroup$
7
$\begingroup$

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: spark.driver.memory 14g

That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above: spark.driver.maxResultSize 2g

Improve this answer
$\endgroup$
2
  • $\begingroup$ 14g is not a lot??? It's not big data but it is actually a lot! $\endgroup$
    – eliasah
    Oct 23, 2015 at 18:26
  • $\begingroup$ Great answer and the only that worked for me. Thanks. $\endgroup$
    – Francesco Boi
    Jan 9, 2019 at 14:49
4
$\begingroup$

Just use the config option when setting SparkSession (as of 2.4)

MAX_MEMORY = "5g"

spark = SparkSession \
    .builder \
    .appName("Foo") \
    .config("spark.executor.memory", MAX_MEMORY) \
    .config("spark.driver.memory", MAX_MEMORY) \
    .getOrCreate()
Improve this answer
$\endgroup$
3
  • $\begingroup$ i get the error :This SparkContext may be an existing one. $\endgroup$
    – Arash
    Jul 20, 2019 at 13:13
  • $\begingroup$ Just replace you code by this one, instead of adding it. The message said that you already created one session, $\endgroup$
    – LaSul
    Jul 21, 2019 at 12:05
  • $\begingroup$ spark.stop() also useful if spark is running at the time $\endgroup$
    – Chris Huang-Leaver
    Sep 8 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.