Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [batch-normalization]

For questions about Batch Normalization of layer activations in theory and practice, as used in (typically deep) neural networks.

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Can principal components changed by a normaliaation method be used to construct original data shape with SVD

I would like to use an algorithm called Harmony to normalize my data. Harmony takes as input principal components ($PC$), and ...
MadmanLee's user avatar
  • 101
7 votes
1 answer
660 views

Why Batch Normalization is undesirable?

In Let's build GPT: from scratch, in code, spelled out., Andrej Karpathy says that no one likes Batch Normalization layer and people want to remove it. He also said it brings so many bugs and he shot ...
mon's user avatar
  • 721
0 votes
1 answer
49 views

batch Normalization and Layer Normalization difference

In Batch Normalization, mean and standard deviation are calculated feature wise and normalization step is done instance wise and in Layer Normalization mean and standard deviation are calculated ...
April's user avatar
  • 1
0 votes
2 answers
102 views

Dropout and BatchNorm decrease speed of learning

Experimenting with the cifar10 dataset and faced with strange behavior when Dropout and BatchNorm don't help at all. As I get: Dropout - freezing some of the weights which helps us to prevent ...
kirsanv43's user avatar
  • 103
0 votes
0 answers
13 views

Why do we use Gradient Descent for Norm Batch since the function has only 2 parameters?

So here is the formula for Z sedile ( gamma * Z + Beta). Since it's a function of 2 parameters why there is still stated in courses that we need to compute gradient descent to find the slope ? Since ...
Lazu Razvan's user avatar
1 vote
2 answers
242 views

Should we always use Batch Renormalization over Batch Normalization?

According to a machine level mastery post on batch norm For small mini-batch sizes or mini-batches that do not contain a representative distribution of examples from the training dataset, the ...
RAbraham's user avatar
  • 187
1 vote
1 answer
46 views

How does batch normalization make a model less sensitive to hyperparameter tuning?

Question 22 of 100+ Data Science Interview Questions and Answers for 2022 asks What is the benefit of batch normalization? The first bullet of the answers to this is The model is less sensitive to ...
Galen's user avatar
  • 172
0 votes
1 answer
81 views

Why doesn't batch normalization 'zero out' a batch of size one?

I'm using Tensorflow. Consider the example below: ...
worduser's user avatar
0 votes
1 answer
465 views

Equations in "Batch normalization: theory and how to use it with Tensorflow"

I read the article Batch normalization: theory and how to use it with Tensorflow by Federico Peccia. The batch normalized activation is $$ \bar x_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} $$...
Triceratops's user avatar
2 votes
1 answer
360 views

Explanation of Karpathy tweet about common mistakes. #5: "you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm"

I recently found this twitter thread from Andrej Karpathy. In it he states a few common mistakes during the development of a neural network. you didn't try to overfit a single batch first. you forgot ...
KDecker's user avatar
  • 123
2 votes
1 answer
2k views

To freeze or not, batch normalisation in ResNet when transfer learning

I'm using a ResNet50 model pretrained on ImageNet, to do transfer learning, fitting an image classification task. The easy way of doing this is simply freezing the conv layers (or really all layers ...
amateurjustin's user avatar
0 votes
1 answer
76 views

Batch normalization for multiple datasets?

I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data. I was told that batch ...
Manveru's user avatar
  • 111
1 vote
1 answer
86 views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
AlexM's user avatar
  • 23
3 votes
1 answer
6k views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
user3668129's user avatar
1 vote
1 answer
235 views

Poor CNN performance after implementing BatchNormalization

I am training a CNN to classify malware images from a dataset named Malimg. Before implementing the BatchNormalization layer, I was getting an accuracy of 95.57% (see below for the graph of loss/...
Jack's user avatar
  • 155
1 vote
1 answer
259 views

Can Batch Normalization replace tanh in RNN?

Question Can Batch Normalization (BN) be inserted in RNN after $x_t@W_{xh}$, and after $h_{t-1}@W_{hh}$ to remove $f=tanh$ and bias $b_h$? If possible, will this ...
mon's user avatar
  • 721
1 vote
0 answers
66 views

BUG: Tensorflow accuracy on training data different when training than when evaluating

I think I have made some sort of error because my model has a substantially different accuracy while training: ...
user3808430's user avatar
1 vote
0 answers
369 views

How to properly train batch-normalization networks with gradient accumulation?

Using gradient accumulation efficiently replicate training with larger batch sizes for networks that are independent on the batch size. However, networks with BN layers rely on a batch size -- running ...
Mikhail Kotyushev's user avatar
4 votes
1 answer
1k views

How does batch normalization work for convolutional neural networks

I am trying to understand how batch normalization (BN) works in CNNs. Suppose I have a feature map tensor $T$ of shape $(N, C, H, W)$ where $N$ is the mini-batch size, $C$ is the number of channels, ...
IntegrateThis's user avatar
2 votes
1 answer
2k views

Batch normalization for image CNN - Why not use the mean of the entire batch?

Question For CNN to recognize images, why not use the entire batch data, instead of per feature, to calculate the mean in the Batch Normalization? When each feature is independent, need to use per ...
mon's user avatar
  • 721
5 votes
1 answer
2k views

Transfer Learning for CNNs and Batch Norm Layers

In some transfer learning models, we set the training argument to False to maintain the pre-trained values of Batch Normalization for example, but the ...
Jack Armstrong's user avatar
1 vote
1 answer
419 views

Why does batchnorm1d in Pytorch compute 0 with the following example (2 lines of code)?

Here is the code ...
Sin Nombre's user avatar
1 vote
0 answers
127 views

Normalizing historical data in time-series LSTMs

I am currently trying to solve a sequence prediction problem using LSTMs in a keras architecture. To illustrate the problem I give the following example which resemble the problem I must solve. Lets ...
seeiespi's user avatar
  • 131
1 vote
0 answers
277 views

How to standardise/normalise csv data in a tensorflow 2 dataset

Does anyone know how to apply standardisation (or normalisation) on the numerical features of a dataset loaded in batch via tf.data.experimental.make_csv_dataset in ...
Tominator's user avatar
  • 111
3 votes
1 answer
2k views

Keras BatchNormalization axis

I use spectrogram as input to a Convolutional Neural Network I have created with tensorflow.keras in Python. Its shape is (time, frequency, 1). The input's shape of the CNN is (None, time, ...
E. Vasilopoulos's user avatar
1 vote
1 answer
867 views

Which batch size to use when Batch Normalization?

I want to train a CNN in Keras (optimizer Adam) and by using batch normalization after every ConvLayer and before every activation layer. So far I mostly see examples in which training is carried out ...
Code Now's user avatar
  • 403
2 votes
3 answers
801 views

Normalization in production

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new ...
Nick's user avatar
  • 21
1 vote
1 answer
2k views

Why would batch normalization allows us to use higher learning rate in the neural network?

I am doing some study about the BatchNormalization: https://towardsdatascience.com/batch-normalization-8a2e585775c9 In the article, it says: ...
Edamame's user avatar
  • 2,755
0 votes
1 answer
5k views

The added layer must be an instance of class Layer. Found: <keras.layers.normalization.BatchNormalization object at 0x0000024B0C16B780>

I don't know why BatchNormalization is giving the following error. Googled out but couldn't find any relevant answers My code appears to be this:
Alvi Rahman's user avatar
1 vote
0 answers
143 views

BatchNorm vs InstNorm from the perspective of feature distributions

What I understand so far... The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift". Covariance shift is ...
AgnosticCucumber's user avatar
1 vote
1 answer
897 views

Batch Normalization with CUDNN

I want to introduce Batch Normalization in my C++/CUDNN implementation of CNN. The implementation is currently performing well (without BN) on the MNIST dataset. I am using the CUDNN implementation ...
Victor Huerlimann's user avatar
1 vote
0 answers
82 views

L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks

Could anyone help to derive equation (15) dL/dx in L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks ? I found that the term inside the rectangle for the expression (12) is ...
kevin's user avatar
  • 283
1 vote
1 answer
718 views

Batch Normalization and Dropout together causing incorrect segmentation results

So, I've been running a test to see how well a number of networks can perform road segmentation on a particular customer's dataset. I am testing UNET, RDRCNN, and Tiramisu against each other. UNET ...
Jake Fleisig's user avatar
2 votes
1 answer
166 views

Where can Batch Normalization be used? CNNs or everywhere?

Should BatchNormalization be used only in CNNs or can they be used in Fully Connected Networks, Recurrent networks as well?
Nagabhushan S N's user avatar
1 vote
1 answer
519 views

How does Batch Normalization in Machine Learning address covariate shift and speed up training?

In this video and this answer, it's mentioned that batch normalization doesn't allow the mean and variance of the parameters of any particular hidden layer to vary too much with change in previous ...
Shirish's user avatar
  • 299
1 vote
0 answers
173 views

Why does BatchNorm on FC layers increase my error?

I am training a deep CNN for multivariate regression, with three fully connected layers on top of the convolutions. I am using Sigmoid activation for FC layers. When adding BatchNormalization (BN) I ...
Camill Trüeb's user avatar
2 votes
2 answers
2k views

Keras models break when I add batch normalization

I'm creating the model for a DDPG agent (keras-rl version) but i'm having some trouble with errors whenever I try adding in batch normalization in the first of two networks. Here is the creation ...
axon's user avatar
  • 23
1 vote
2 answers
5k views

When to use model.train() vs model.eval() in Pytoch?

I have a model that is used in a reinforcement learning algorithm for checkers, a la AlphaZero. Similar to that network, mine features batch normalization after each convolution layer. I am aware that ...
basket's user avatar
  • 157
1 vote
1 answer
1k views

What exactly is BatchNormalization() in Keras?

A month or two straight away into building image classifiers, I just sandwiched the BatchNormalization layer between conv2d. I wonder what it does, but I have seen my model learn faster in presence of ...
Nikhil.Nixel's user avatar
0 votes
1 answer
352 views

Compute gradients in parallel

Here is part of my code: ...
Alex Marshall's user avatar
0 votes
0 answers
211 views

Neural Network fails to train when a particular batch normalization layer is removed?

Background info: I built a baseline CNN model for the cifar10 dataset using batch normalization and then ReLu activation after each convolution layer. There are a couple of max pooling layers in ...
valentinocc's user avatar
1 vote
1 answer
2k views

If we are using batch normalization as the first layer, can we forego standard scaling of inputs?

It is common practice to use the standard scaler on the inputs before feeding it to a deep learning architecture. I was wondering whether it is necessary if the first layer is a batch normalization ...
Luv's user avatar
  • 131
4 votes
1 answer
2k views

Keras multi-gpu batch normalization

1) How does the batch normalization layer work with multi_gpu_model? Is it calculated separately on each GPU, or is somehow synchronized between GPUs? 2) Which batch normalization parameters are ...
Antonio Jurić's user avatar
1 vote
0 answers
317 views

Do batch norm makes sense for regression problems?

When my network is performing regression (like DQN) it makes sense to use batchnorm in network when output of my network should vary from [0, 100000]? one way to tackle it is to normalize output but ...
quester's user avatar
  • 295
1 vote
0 answers
38 views

How much batch effect is too much batch effect?

Algorithms such as ComBat/SVA are powerful tools for the removal of batch effects. Small batch effects can be confidently removed by these methods. But surely there must exist batch effects which are ...
lars20070's user avatar
1 vote
2 answers
2k views

Conv bias or not with Instance Normalization?

It is well known that Conv layers that are followed by BatchNorm ones should not have bias due to BatchNorm having a bias term. Using InstanceNorm however, the statistics are instance-specific rather ...
Roulbacha's user avatar
  • 131
0 votes
2 answers
1k views

Why BatchNormalization fails in Keras

I try to test ResNet approach on cifar10 dataset with the following python code: ...
Serge P.'s user avatar
  • 217
11 votes
2 answers
14k views

Batch normalization vs batch size

I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $64$ to $256$. I have also observed that, using batch size $64$, ...
Arka Mallick's user avatar
11 votes
2 answers
4k views

Does batch normalization mean that sigmoids work better than ReLUs?

Batch normalization and ReLUs are both solutions to the vanishing gradient problem. If we're using batch normalization, should we then use sigmoids? Or are there features of ReLUs that make them ...
generic_user's user avatar
3 votes
1 answer
217 views

In batch normalization, shouldn't using DropConnect harm test accuracy?

In my understanding of batch normalization, mean and variance are calculated over the entire batch and then added to the population average. This average is then applied to the test set to estimate ...
Random User's user avatar