# Different learning rates for each dimension

I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent.

One thing that got me wondering is that we use a pre-defined set of learning rates for each of our dimensions. Now I believe if we don't normalize them we would need to keep our learning rate small due to different scales of the features, but that got me thinking, can't we have different learning rates (suited to each feature dimension scale) to compensate for not doing the normalization?

The answer to your question is yes, and it is already implemented due to popular approaches which are introduced in academic papers. For instance, Adam optimisaiton approach is an optimiser which assumes it is not valid to multiply each dimension, feature, with a specified value. It uses the idea of momentum and geometric average and a bunch of other methods to employ the idea that you should not go down the hill with the same speed for different dimensions. What it says is that you may be in a location which each dimension has different slope, and due to this fact you should not go downhill with the same learning rate. In that approach, although you specify the same learning rate for the optimiser, due to using momentum, it changes in practice for different dimensions. At least as far as I know, the idea of different learning rates for each dimension was introduced by Pr. Hinton with his approache, namely RMSProp.

Welcome to the group :).

Before answering your question, it will be good if I explain why and how "learning rate" is used. For that sharing an equation below: Here theta's are weight and alpha is learning rate.

This equation is Gradient Descent equation, used to optimize weights. Internally, optimizer generally perform similar type of equation for each weight(weights associated with different feature) separately.

Learning rate value describes, how much adjustment should it made in previous weights. Higher the value, faster it converge(but may skip out the best value, thats why optimal value is used).

Now going back to your question: " can't we have different learning rates (suited to each feature dimension scale) to compensate for not doing the normalization?"

If I consider mathematically, according to me yes we can use different learning rate as per the feature values and we surely give a try in evaluating features weight.

So far I know about Tensorflow and Scikit, both use a single learning rate and its is generic for all features. So if you want to use different learning rate(per feature), either you have to write your own optimizer code or use some other library(not sure which one).

Additional Note may be helpful: Definitely feature scaling helps in faster convergence in terms of feature weight calculations. But if we dont do feature scaling, in case of some algo's like KNN, K-Means features value may influence the model training as well.

Short Answer - Yes. Adam-bias-corrected is a good example with all benefits.

Long answer- RMSProp by Hinton & Adadelta ( with corrected units by Hessian approximatin), both methods do same what you have asked for ie doensn't work on single learning rate, rather have different learning rates and also they don't need a selection of global learning rate ( which is the case with original Adadelta). Both of them work in similar fashion with their own pros and cons. Adgrad is the first one which was introduced with adaptive learning rates for different paramters. while RMSprop has exponentially decaying average which stores this decaying average of past squared gradients vt. mt below is the past gradients' average which brings us to Adam - bias-correct method. ABove, since mt and vt are initialised as vectors of zeros, they are biased towards zero, expecially during the initial time steps or when the decay rates are small, hence Adam - bias -correct which is works with corrected values as follow Hope that helps.