Hashing trick with random forest in scala

Question

I am trying to perform a hashing trick and then a random forest with scala. I have the following code:

val documents: RDD[Seq[String]] = sc.textFile("hdfs:///tmp/new_cromosoma12v2.csv").map(_.split(",").toSeq)

val hashingTF = new HashingTF()
val tf: RDD[Vector] = hashingTF.transform(documents)

val splits = tf.randomSplit(Array(0.7, 0.3))
val (trainingData, testData) = (splits(0), splits(1))

val numClasses = 3
val categoricalFeaturesInfo = Map[Int, Int]()
val numTrees = 10 
val featureSubsetStrategy = "auto" 
val impurity = "gini"
val maxDepth = 8
val maxBins = 32

**val trainingData2=LabeledPoint(1.0,trainingData.collect())**

val model = RandomForest.trainClassifier(trainingData2, numClasses, categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)

But I have the error

found : Array[org.apache.spark.mllib.linalg.Vector] required: org.apache.spark.mllib.linalg.Vector

in the bold line.

Do you know how I can solve it?

Thank you,

Laia

Answer 1

Let's look at the error message:

found : Array[org.apache.spark.mllib.linalg.Vector] required: org.apache.spark.mllib.linalg.Vector

"found" is the type of object that it came across, "required" is the type of the object that the function accepts. The types look mostly the same ( org.apache.spark.mllib.linalg.Vector ) but the first is Array[X] and the second is X.

Why? Because trainingData.collect() gives you the RDD as an array of objects in the array.

What you probably want to do instead is map, to turn each vector into a LabeledPoint, instead of trying to turn the whole RDD into a single LabeledPoint.

Answer 2

LabeledPoint expects: LabeledPoint(label: Double, features: Vector)

when you execute val trainingData2=LabeledPoint(1.0,trainingData.collect()) you're actually getting all the Rows in your trainingData set so you will have Array(Row(???), Row(???), ???) but what you need is to apply def transformToLabeledPoint(vector: Vector) = LabeledPoint(1.0, vector), you can either apply a map on your trainingData or create a UDF and apply it to your trainingData, for instance:

import org.apache.spark.ml.feature.SQLTransformer
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors

val df = spark.createDataFrame(Seq((1.0,100.0),(2.0,200.0))).toDF("id", "value")

spark.udf.register("setLabel", (label: Double, x: Seq[Double]) => LabeledPoint(label, Vectors.dense(x.toArray)))

val labeledTrainingData = new SQLTransformer()
    .setStatement("SELECT setLabel(1.0, array(*)) AS points FROM __THIS__")
    .transform(df)
val labeledRDD = labeledTrainingData.rdd.map(_(1).asInstanceOf[LabeledPoint])