Batch normalization for multiple datasets?

Question

I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.

I was told that batch normalization layers might be trying to find weights that are good for all while training, which is a problem since the distribution of my synthetic data is not exactly equal to the distribution of the real data. So, the idea would be to have different 'copies' of the weights of batch normalization layers. So that the neural network estimates different weights for synthetic and real data, and uses just the weights of real data for evaluation.

My question is, how to perform batch normalization in the aforementioned case? Is there already an implementation of batch norm layers in PyTorch that solves the problem?

Answer 1

You can normalize your data manually by setting absolute min and max values that go beyond the training and real world values.

For instance, if the maximum of a training feature is 1500, and the maximum in real world data is 3000 but it could be 3500 as great maximum in the future, you should take 3500 for the training.

absolute_max_A = 3500

training_data_A = [50, 1500, 600]

normalized_training_data_A = [x / absolute_max_A for x in training_data_A] 

normalized_training_data_A

#[0.014285714285714285, 0.42857142857142855, 0.17142857142857143]