Should I consider feature scaling for all gradient descent based algorithms?

Question

In the Coursera course machine learning in the section on Multivariate Linear Regression, Andrew Ng provides the following tips on gradient descent:

• Use Feature Scaling to converge quicker
  • Get feature into an approx -1 < x < 1 range
  • Mean normalization

Andrew Ng also provides some other tips:

• Plot cost vs iterations
  • to ensure cost decreases on every iteration (try smaller alpha)
  • to identify if convergence is too slow (try larger alpha)
  • to identify approximately the number of iterations to converge

Are these tips applicable to all problems involving gradient descent using different machine/deep learning algorithms or just to Multivariate Linear Regression?

Answer 1

About the tips regarding plot cost vs. iteration, they are generally applicable to gradient descent approaches, including deep learning, where hyperparameter tuning (e.g. learning rate) is crucially important.

About the proper input scaling, it is not only related to the machine learning approach, but to the specific problem under consideration. Sometimes machine learning algorithms rely on distances to compute the similarity between individuals. Scaling changes some of these distances. In these cases, the resulting distance after scaling should be assessed to check whether it is more appropriate than without scaling. Here you can find examples for clustering. For some machine learning algorithms, you need standardized features, e.g. regularized linear/logistic regression. For most optimization-based machine learning algorithms, it makes sense to have feature scaling. On the other hand, there are problems where scaling doesn't even make sense (e.g. discrete input problems, like token-based natural language processing).