3
$\begingroup$

I am thinking of preprocessing techniques for the input data to a convolutional neural network (CNN) using sparse datasets and trained with SGD. In Andrew Ng's coursera course, Machine Learning, he states that it is important to preprocess the data so it fits into the interval $ \left[ 3, 3 \right] $ when using SGD. However, the most common preprocessing technique is to standardize each feature so $ \mu = 0 $ and $ \sigma = 1 $. When standardizing a highly sparse dataset many of the values will not end up in the interval.

I am therefore curious - would it be better to aim for e.g. $ \mu = 0 $ and $ \sigma = 0.5 $ in order for the values be closer to the interval $ \left[ 3, 3 \right] $? Could anyone argue based on a knowledge of SGD on whether it is most important to aim for $ \mu = 0 $ and $ \sigma = 1 $ or $ \left[ 3, 3 \right] $?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

No, you are misinterpreting his comments. If you have data that has some outliers in it then the outliers will extend beyond 3 standard deviations. Then if you standardize the data some will extend beyond the [-3,3] region.

He is simply saying that you need to remove your outliers so the outliers don't reap havoc on your stochastic gradient descent algorithm. He is NOT saying that you need to use some weird scaling algorithm.

You should standardize your data by subtracting the mean and dividing by the standard deviation, and then remove any points that extend beyond [-3,3], which are the outliers.

In stochastic gradient descent, the presence of outliers could increase the instability of the minimization and make it thrash around excessively, so its best to remove them.

If the sparseness of the data prevents removal then... Do you need to use stochastic gradient descent, or can you just use gradient descent? Gradient descent (GD) might help to alleviate some of the problems relating to convergence. Finally, if GD is having trouble converging, you could always do an direct solve (e.g. direct matrix inversion) rather than an iterative solve.

Hope this helps!

Improve this answer
$\endgroup$
5
  • $\begingroup$ Neither GD or direct solve is feasible as it would be take too many computational resources when training a convolutional neural network (CNN). While it therefore did not solve my problem, your answer provided a nice insight to the issue. $\endgroup$
    – pir
    Commented Jul 11, 2015 at 22:18
  • $\begingroup$ Wouldn't GD also be affected by outliers, although to a lesser degree? $\endgroup$
    – pir
    Commented Jul 11, 2015 at 22:22
  • $\begingroup$ Affected by, yes, but to a lesser degree i.e. less stochastic. In addition to just turning down the learning coefficient, you can prime lots of descenders and use the most common minima. $\endgroup$
    – AN6U5
    Commented Jul 12, 2015 at 0:32
  • $\begingroup$ How can you simultaneously have a sparse dataset for which removing outliers isn't feasible and a problem with so much data that resources are tight and SGD is needed? Is it that you have a non normal dataset? Then try some normalizing transformations like log, square root, or inverse hyperbolic tangent. $\endgroup$
    – AN6U5
    Commented Jul 12, 2015 at 2:15
  • $\begingroup$ The ML course doesn't cover mini-batches, which would help here as a compromised between fully online SGD and full batch learning. Also using dropout for regularisation may help. $\endgroup$
    – Neil Slater
    Commented Jul 12, 2015 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.