3
$\begingroup$

I have been trying to run linear regression with SGD that is found in Spark mllib for some time and I am experiencing huge performance problems. All examples that I was looking have number of iterations set to 100 and say that Spark mllib can run very fast for big data.

This the code I am having problems with:

    def train(data: RDD[TrainFeature], scalerModel: StandardScalerModel): LinearRegressionModel = {
    val labeledData = data map { trainFeature =>
      LabeledPoint(trainFeature.relevance.value, toVector(trainFeature.feature, scalerModel))
    }
    labeledData.cache()

    val algorithm = new LinearRegressionWithSGD()
    algorithm.optimizer
      .setNumIterations(10)
      .setRegParam(0.01)
      .setStepSize(0.1)

    algorithm run labeledData
  }

private def toVector(feature: Feature, scalerModel: StandardScalerModel): Vector = scalerModel transform toVector(feature)

private def toVector(feature: Feature): Vector = Vectors dense feature.coordinates.toArray

I have scaled the data first and then run the algorithm to train the model. Even when I use 10 iterations it takes around 10 minutes to train model for 70,000 entries with feature vector of size 2. And the results I get are not good at all. I start getting decent results after numberOfIterations = 1000, but that would take ages.

Is it normal for linear regression with SGD to be this slow for 70,000 vectors of size 2?

My JVM min and initial memory is set to 4g. I have tried setting the following as well (devastated try): System.setProperty("spark.executor.memory", "3g")

I am running this locally and since normal LinearRegression written in MatLab would finish the job very fast, I am wondering what am I doing wrong?

Edit: When I look at the spark UI in jobs section, I see that it is creating way too many jobs for gradient descent. Is there a way that I could tell Spark to create very little jobs - i.e. don't split data, run everything in a single thread? Maybe that can help me debug the problem further.

Improve this question
$\endgroup$
7
  • $\begingroup$ It still seems very slow, but what do you mean by regular linear regression? Comparing regular linear regression with L2 loss function to SGD methods is unfair, L2 loss function linear regression does not have to iterate, it has a closed form solution. $\endgroup$
    – Jan van der Vegt
    Commented Feb 29, 2016 at 10:17
  • $\begingroup$ Have you excluded all other computational tasks like scaler etc. from your pipeline measurement loop? $\endgroup$
    – Diego
    Commented Mar 1, 2016 at 0:07
  • $\begingroup$ Yes, I have excluded all other computational tasks. Also, it seems to work faster when I use only one split for flie reading - I am reading data from CSV. @JanvanderVegt I was comparing it with linear regression that I have written myself. It is using simple gradient descent. I honestly don't know how SGD works so maybe it is supposed to be this slow - but I feel like it is unlikely since 70,000k entries should be processed in 5 minutes for 100 steps. $\endgroup$
    – Nikola Stojiljkovic
    Commented Mar 1, 2016 at 11:25
  • 3
    $\begingroup$ Why are you using spark for 70k rows? That really doesn't make any sense and will likely require more time and result in a worse model than using a simple in-memory algorithm. To address your "edit", try using fewer partitions on your training data, you can do this via coalesce. $\endgroup$
    – David
    Commented Mar 1, 2016 at 17:39
  • 1
    $\begingroup$ Number of jobs is related to the number of iterations you set. It can be lower if SGD converges before reaching NumIterations but it cannot be lowered manually. Regarding performance - it takes less than a fives seconds (~160 iterations) on a few years old box with random data of the same shape as yours. It would be useful if you show how you load and prepare the data. $\endgroup$
    – zero323
    Commented Mar 5, 2016 at 11:27

1 Answer 1

Reset to default
0
$\begingroup$

Spark is designed to be distributed across a cluster and use stochastic gradient descent (SGD) to optimize linear regression.

There is overhead for cluster infrastructure (even when the "cluster" is a single local node). Also, SGD is an iterative method that uses many batches to find a solution.

Given that your problem is 70k rows, it would be better to use a single node framework (e.g., scikit-learn) and ordinary least squares (OLS), a closed form solution, to optimize linear regression. Those two changes will greatly speed up training.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.