# How to do an incremental update for the mean and standard deviation of tensor data?

I have a big dataset (some 400Gb) consisting of tensor data (shape is $$(600, 600, 10)$$) and I want to normalize this dataset before feeding it to a neural network but this dataset can't fit in my memory so I was wondering about incremental updates of the mean and standard deviation.

The formulas are here but I don't see how to adapt them in the tensor case since I don't have only one value per new iteration step but a whole tensor, and I can't take the mean of this new tensor since the mean of means won't be the mean of the whole dataset.

Normalization is needed to make each input feature vary within the same scale around the same mean. If one feature varies within a much larger scale, it will have a larger influence on the error that your network minimizes during training. For example, this explanation: link.

Your input is a collection of data in the shape $$(600,600,10)$$. Each of those $$600 \times 600 \times 10$$ inputs is a separate input feature. To normalize them, you need to apply this formula to each of your $$600 \times 600 \times 10$$ inputs separately, and thus get $$600 \times 600 \times 10$$ means and $$600 \times 600 \times 10$$ standard deviations. Then subtract each mean from the corresponding input, and divide it by the corresponding standard deviation.

However, you might not want to do this, especially if there is some internal structure within your $$(600,600,10)$$ tensor which you may want to preserve after the normalization. The way you normalize will then depend on which property you want to preserve.

For example, during the simple normalization described above the values that are close to each other will be mapped to values that are still close, and the values that are very different will be mapped to values that are far apart. If this is enough, then you can use this normalization. However, the inputs that were equal before the normalization may be mapped to different values as each will be normalized differently. This property will be destroyed, and if you want to keep it, you need to normalize differently.

For example, in an image, which is a $$(n,m,3)$$ tensor of pixels, each pixel in a certain contour may have the same value. You want to these pixels to have the same value after the normalization to detect this contour, so you cannot normalize each pixel individually. Then you may want to scale all your inputs uniformly, that is map all pixel values between 0 and 255 to $$[0, 1]$$ or $$[-1, 1]$$.

If you want your data to be normally distributed and mapped uniformly, then you need to consider every $$(600,600,10)$$ tensor just as a linear sequence of points, combine all your tensors into one long linear sequence, find the mean, and standard deviation of this sequence, and then subtract this mean from each input, and divide it by the standard deviation.

• Thank you a lot for your answer. At first I wanted to go with $600x600x10$ means and standard deviations but I thought that it was still a lot. There's definitely some internal structure I want to preserve so I'll check how good the uniform mapping to $[-1, 1]$ works. I can't upvote your question, I need 15 reputation but it said they got my feedback. Thanks again. Commented Oct 5, 2022 at 10:58