Any differences in regularisation in MLP between batch and individual updates?

Question

I have just learned about regularisation as an approach to control over-fitting, and I would like to incorporate the idea into a simple implementation of backpropagation and Multilayer perceptron (MLP) that I put together.

Currently to avoid over-fitting, I cross-validate and keep the network with best score so far on the validation set. This works OK, but adding regularisation would benefit me in that correct choice of the regularisation algorithm and parameter would make my network converge on a non-overfit model more systematically.

The formula I have for the update term (from Coursera ML course) is stated as a batch update e.g. for each weight, after summing all the applicable deltas for the whole training set from error propagation, an adjustment of lambda * current_weight is added as well before the combined delta is subtracted at the end of the batch, where lambda is the regularisation parameter.

My implementation of backpropagation uses per-item weight updates. I am concerned that I cannot just copy the batch approach, although it looks OK intuitively to me. Does a smaller regularisation term per item work just as well?

For instance lambda * current_weight / N where N is size of training set - at first glance this looks reasonable. I could not find anything on the subject though, and I wonder if that is because regularisation does not work as well with a per-item update, or even goes under a different name or altered formula.

Answer 1

Regularization is relevant in per-item learning as well. I would suggest to start with a basic validation approach for finding out lambda, whether you are doing batch or per-item learning. This is the easiest and safest approach. Try manually with a number of different values. e.g. 0.001. 0.003, 0.01, 0.03, 0.1 etc. and see how your validation set behaves. Later on you may automate this process by introducing a linear or local search method.

As a side note, I believe the value of lambda should be considered in relation to the updates of the parameter vector, rather than the training set size. For batch training you have one parameter update per dataset pass, while for online one update per sample (regardless of the training set size).

I recently stumbled upon this Crossvalidated Question, which seems quite similar to yours. There is a link to a paper about a new SGD algorithm, with some relevant content. It might be useful to take a look (especially pages 1742-1743).

Answer 2

To complement what insys said :

Regularization is used when computing the backpropagation for all the weights in your MLP. Therefore, instead of computing the gradient in regard to all the input of the training set (batch) you only use some/one item(s) (stochastic or semi-stochastic). You will end up limiting a result of the update in regard to one item instead of all which is also correct.

Also, if i remember correctly, Andrew NG used L2-regularization. The /N in lambda * current_weight / N is not mandatory, it just helps rescaling the input. However if you choose not to use it, you will have (in most of the case) to select another value for lambda.

You can also use the Grid-search algorithm to choose the best value for lambda (the hyperparameter => the one you have to choose).