# Task not serializable Error

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.cassandra.CassandraSQLContext
object Test {
val sparkConf = new SparkConf(true).set("spark.cassandra.connection.host", <Cassandra Server IP>)
val sc = new SparkContext(sparkConf)
val cassandraSQLContext = new CassandraSQLContext(sc)
val numberAsString = cassandraSQLContext.sql("select * from testing.test").first().getAs[Int]("number").toString()
val testRDD = sc.parallelize(List(0, 0))
val newRDD = testRDD.map { x => numberAsString }
}


This is the code that I have written in Spark. I was expecting it to work properly because i am evaluating the numberAsString value and then using it in the map function but it gives me task not serializable error. I am running the job in local mode.

• I looked at the answer by @PierceHardman and I got the answer to my problem. – Credosam Sep 15 '16 at 6:54

Looking at another question on Stack Overflow about serialization exceptions in Spark, it says that anonymous functions serialize their containing class, and if that class contains the SparkContext -- which is not serializable -- then an error is thrown.

Perhaps this is happening to you?

I changed my code to this:

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.cassandra.CassandraSQLContext
import org.apache.spark.rdd.RDD
object Test2 {
def calculate(numberAsString: String, testRDD: RDD[Int]): RDD[String] = {
val newRDD = testRDD.map { x => numberAsString }
newRDD
}
}

object Test {
val sparkConf = new SparkConf(true).set("spark.cassandra.connection.host", <Cassandra Server IP>)
val sc = new SparkContext(sparkConf)
val cassandraSQLContext = new CassandraSQLContext(sc)
val numberAsString = cassandraSQLContext.sql("select * from hdfc.test").first().getAs[Int]("number").toString()
val testRDD = sc.parallelize(List(0, 0))
val newRDD = Test2.calculate(numberAsString, testRDD)
}


Now when it will go to serialize the containing class then it will get serialized.

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.cassandra.CassandraSQLContext
object Test {
val sparkConf = new SparkConf(true).set("spark.cassandra.connection.host", <Cassandra Server IP>)
val sc = new SparkContext(sparkConf)
val cassandraSQLContext = new CassandraSQLContext(sc)
val numberAsString = cassandraSQLContext.sql("select * from testing.test").first().getAs[Int]("number").toString()
val numberAsStringBC = sc.broadcast(numberAsString)
val testRDD = sc.parallelize(List(0, 0))
val newRDD = testRDD.map { x => numberAsStringBC.value }
}

• I tried broadcasting the variable. There is no change in the error. It still shows me Task not serializable error. I am running the job in local mode. – Credosam Sep 14 '16 at 13:47