My ultimate goal is to use Jupyter together with Python for data analysis using Spark. The current hurdle I face is loading the external
spark_csv library. I am using Mac OS and Anaconda as the Python distribution.
In particular, the following:
from pyspark import SparkContext sc = SparkContext('local', 'pyspark') sqlContext = SQLContext(sc) df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv') df.show()
when invoked from Jupyter yields:
Py4JJavaError: An error occurred while calling o22.load. : java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org
Here are more details:
Setting Spark together with Jupyter
I managed to set up Spark/PySpark in Jupyter/IPython (using Python 3.x).
System initial setting
On my OS X I installed Python using Anaconda. The default version of Python I have currently installed is 3.4.4 (Anaconda 2.4.0). Note, that I also have installed also 2.x version of Python using
conda create -n python2 python=2.7.
This is actually the simplest step; download the latest binaries into
~/Applications or some other directory of your choice. Next, untar the archive
tar -xzf spark-X.Y.Z-bin-hadoopX.Y.tgz.
For easy access to Spark create a symbolic link to the Spark:
ln -s ~/Applications/spark-X.Y.Z-bin-hadoopX.Y ~/Applications/spark
Lastly, add the Spark symbolic link to the PATH:
export SPARK_HOME=~/Applications/spark export PATH=$SPARK_HOME/bin:$PATH
You can now run Spark/PySpark locally: simply invoke
In order to use Spark from within a Jupyter notebook, prepand the following to
Further details can be found here.