1
$\begingroup$

I try to create a machine learning model, linear regression, to predict a price of a diamonds.

All examples that I found online do not have a step with scaling of data, using MinMaxScaler or StandardScaler. But I personally think that this is an important step in ML. The Spark guide shows this feature, but separately from OneHotEncoding. Moreover, because the result of OneHotEncoding in Scala is different from Python (for instance, it looks like this: (4,[3],[1.0]) in Spark, while in Python it is a combination of 1 and 0), I am confused where to apply StandardScaler - after Indexing and before OneHotEncoder, or after OneHotEncoder or in some other step?

My code after I cleaned data a little bit. This code works, but predictions are far from being good. I use Scala dataframe and pipeline:

import org.apache.spark.ml.feature.{VectorAssembler,StringIndexer,VectorIndexer,OneHotEncoder}
import org.apache.spark.ml.linalg.Vectors

val cutIndexer = new StringIndexer().setInputCol("cut").setOutputCol("cutIndex")
val colorIndexer = new StringIndexer().setInputCol("color").setOutputCol("colorIndex")
val clarityIndexer = new StringIndexer().setInputCol("clarity").setOutputCol("clarityIndex")

val cutEncoder = new OneHotEncoder().setInputCol("cutIndex").setOutputCol("cutVec")
val colorEncoder = new OneHotEncoder().setInputCol("colorIndex").setOutputCol("colorVec")
val clarityEncoder = new OneHotEncoder().setInputCol("clarityIndex").setOutputCol("clarityVec")

val assembler = (new VectorAssembler()
                  .setInputCols(Array("carat", "cutVec", "colorVec", "clarityVec", "depth", "table", "x", "y", "z"))
                  .setOutputCol("features") )

val scaler = new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithStd(true).setWithMean(false)

val Array(training, test) = df_label.randomSplit(Array(0.75, 0.25))

import org.apache.spark.ml.Pipeline

val lr = new LinearRegression()

val pipeline = new Pipeline().setStages(Array(cutIndexer,colorIndexer, clarityIndexer,cutEncoder,colorEncoder,clarityEncoder, assembler, scaler, lr))

val model = pipeline.fit(training)

val results = model.transform(test)

The subset of data:

+-----+-----+---------+-----+-------+-----+-----+----+----+----+
|label|carat|      cut|color|clarity|depth|table|   x|   y|   z|
+-----+-----+---------+-----+-------+-----+-----+----+----+----+
|  326| 0.23|    Ideal|    E|    SI2| 61.5| 55.0|3.95|3.98|2.43|
|  326| 0.21|  Premium|    E|    SI1| 59.8| 61.0|3.89|3.84|2.31|
|  327| 0.23|     Good|    E|    VS1| 56.9| 65.0|4.05|4.07|2.31|

Thanks in advance!

$\endgroup$
1
$\begingroup$

Okay, I came up with a solution by myself. Applied OneHotEncoderEstimator that was added recently instead of OneHotEncode on every categorical feature. Then I used MinMaxScaler on features from VectorAssembler.

val cutIndexer = new StringIndexer().setInputCol("cut").setOutputCol("cutIndex")
val colorIndexer = new StringIndexer().setInputCol("color").setOutputCol("colorIndex")
val clarityIndexer = new StringIndexer().setInputCol("clarity").setOutputCol("clarityIndex")

import org.apache.spark.ml.feature.OneHotEncoderEstimator

val encoder = new OneHotEncoderEstimator().setInputCols(Array("cutIndex", "colorIndex", "clarityIndex")).setOutputCols(Array("cutIndexEnc", "colorIndexEnc", "clarityIndexEnc"))


val assembler = (new VectorAssembler()
                    .setInputCols(Array("carat", "cutIndexEnc", "colorIndexEnc", "clarityIndexEnc", "depth", "table", "x", "y", "z"))
                    .setOutputCol("features_assem") )

import org.apache.spark.ml.feature.MinMaxScaler
val scaler = new MinMaxScaler().setInputCol("features_assem").setOutputCol("features")

val Array(training, test) = df_label.randomSplit(Array(0.75, 0.25))

import org.apache.spark.ml.Pipeline

val lr = new LinearRegression()

val pipeline = new Pipeline().setStages(Array(cutIndexer,colorIndexer, clarityIndexer,encoder, assembler,scaler, lr))

val model = pipeline.fit(training)

val predictions = model.transform(test)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.