Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [batch-normalization]

For questions about Batch Normalization of layer activations in theory and practice, as used in (typically deep) neural networks.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

BatchNorm vs InstNorm from the perspective of feature distributions

What I understand so far... The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift". Covariance shift is ...
0
votes
0answers
7 views

Res Net Reduction Block

I am building a ResNet. I have two separate blocks: Cnn block, Reduce block. Cnn block - 1 cnn layer, activation, Batch Normal -> 1 cnn, activation, Batch Normal, so 2 CNN in this block. In Reduce ...
0
votes
0answers
12 views

Why is batch normalization giving worse results?

I read about ann in Tom Mitchell's text on machine learning (4th chapter). In his book, he takes an example to explain hidden layer representations. He takes a 8*3*8 layer. Where input is $$ ...
0
votes
0answers
31 views

Batch Normalization with CUDNN

I want to introduce Batch Normalization in my C++/CUDNN implementation of CNN. The implementation is currently performing well (without BN) on the MNIST dataset. I am using the CUDNN implementation ...
1
vote
0answers
13 views

L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks

Could anyone help to derive equation (15) dL/dx in L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks ? I found that the term inside the rectangle for the expression (12) is ...
1
vote
1answer
62 views

Batch Normalization and Dropout together causing incorrect segmentation results

So, I've been running a test to see how well a number of networks can perform road segmentation on a particular customer's dataset. I am testing UNET, RDRCNN, and Tiramisu against each other. UNET ...
2
votes
1answer
57 views

Where can Batch Normalization be used? CNNs or everywhere?

Should BatchNormalization be used only in CNNs or can they be used in Fully Connected Networks, Recurrent networks as well?
1
vote
1answer
25 views

How does Batch Normalization in Machine Learning address covariate shift and speed up training?

In this video and this answer, it's mentioned that batch normalization doesn't allow the mean and variance of the parameters of any particular hidden layer to vary too much with change in previous ...
0
votes
0answers
28 views

Why does BatchNorm on FC layers increase my error?

I am training a deep CNN for multivariate regression, with three fully connected layers on top of the convolutions. I am using Sigmoid activation for FC layers. When adding BatchNormalization (BN) I ...
1
vote
2answers
149 views

Keras models break when I add batch normalization

I'm creating the model for a DDPG agent (keras-rl version) but i'm having some trouble with errors whenever I try adding in batch normalization in the first of two networks. Here is the creation ...
0
votes
2answers
541 views

When to use model.train() vs model.eval() in Pytoch?

I have a model that is used in a reinforcement learning algorithm for checkers, a la AlphaZero. Similar to that network, mine features batch normalization after each convolution layer. I am aware that ...
0
votes
1answer
130 views

What exactly is BatchNormalization() in keras?

A month or two straight away building image classifiers, I just sandwich the BatchNormalization layer between conv2d. I wonder, what it does, but I have seen my model learn faster in presence of these ...
0
votes
1answer
36 views

Compute gradients in parallel

Here is part of my code: ...
0
votes
0answers
37 views

Neural Network fails to train when a particular batch normalization layer is removed?

Background info: I built a baseline CNN model for the cifar10 dataset using batch normalization and then ReLu activation after each convolution layer. There are a couple of max pooling layers in ...
0
votes
0answers
106 views

Batch Normalization - explicit pixelwise application

I want to apply Batch-Normalization to a CNN, but I have trouble understanding what exactly is happening. Lets say I have 10 images, each image has the size 16x16. I choose the batch size 2. That ...
0
votes
0answers
28 views

Gradient of batchnorm layer

In recent paper about How Does Batch Normalization Help Optimization? by Satunkar et.al. The paper mentions facts about the derivative of a loss function through a batchnorm layer. The paper state ...
0
votes
1answer
173 views

If we are using batch normalization as the first layer, can we forego standard scaling of inputs?

It is common practice to use the standard scaler on the inputs before feeding it to a deep learning architecture. I was wondering whether it is necessary if the first layer is a batch normalization ...
1
vote
0answers
27 views

Why is it necessary in batch normalization to multiply and add a parameter to the result?

How do we decide on which layer we want to add batch normalization. So if we have chosen a layer to apply batch norm to then why don't just normalize it why are we multiplying and scaling it by some ...
0
votes
0answers
106 views

Transpose-CNN with batch normalization

I am new to GAN (generative adversarial networks) and am trying to implement a Transpose-CNN (T-CNN) in Matlab (which is used as the generator in the GAN). I was able to build a Transpose-CNN with ...
2
votes
1answer
702 views

Keras multi-gpu batch normalization

1) How does the batch normalization layer work with multi_gpu_model? Is it calculated separately on each GPU, or is somehow synchronized between GPUs? 2) Which batch normalization parameters are ...
1
vote
0answers
138 views

Do batch norm makes sense for regression problems?

When my network is performing regression (like DQN) it makes sense to use batchnorm in network when output of my network should vary from [0, 100000]? one way to tackle it is to normalize output but ...
1
vote
0answers
29 views

How much batch effect is too much batch effect?

Algorithms such as ComBat/SVA are powerful tools for the removal of batch effects. Small batch effects can be confidently removed by these methods. But surely there must exist batch effects which are ...
0
votes
2answers
293 views

Conv bias or not with Instance Normalization?

It is well known that Conv layers that are followed by BatchNorm ones should not have bias due to BatchNorm having a bias term. Using InstanceNorm however, the statistics are instance-specific rather ...
0
votes
0answers
23 views

How does single image normalization help face recognition model training?

I knew why/how batch image normalization help model training,but How does single image normalization help face recognition model training?
0
votes
1answer
590 views

Why BatchNormalization fails in Keras

I try to test ResNet approach on cifar10 dataset with the following python code: ...
5
votes
2answers
3k views

Batch normalization vs batch size

I have noticed that my performance of VGG 16 network gets better if I increase the batch size from $64$ to $256$. I have also observed that, using batch size $64$, ...
9
votes
2answers
164 views

Does batch normalization mean that sigmoids work better than ReLUs?

Batch normalization and ReLUs are both solutions to the vanishing gradient problem. If we're using batch normalization, should we then use sigmoids? Or are there features of ReLUs that make them ...
3
votes
1answer
133 views

In batch normalization, shouldn't using DropConnect harm test accuracy?

In my understanding of batch normalization, mean and variance are calculated over the entire batch and then added to the population average. This average is then applied to the test set to estimate ...
5
votes
2answers
2k views

Relationship between batch size and the number of neurons in the input layer

Regarding LSTM neural networks, I am unable to understand the relationship between batch size, the number of neurons in the input layer and the number of "variables" or "columns" in the input. (...
2
votes
1answer
744 views

Batch Normalization and Input Normalization in CNN

I build my CNN on Keras, normally in the ImageDataGenerator I saw the rescale = 1. / 255 used to normalize input data (pixel ...
4
votes
2answers
1k views

deriving the gradient of batch normalization

I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'. I present to you the triangle of death (in red), in the ...
2
votes
2answers
339 views

Batch norm: why the initial normalization?

I'm a beginner in NNs and the first thing I don't understand with batch norm is the following 2 steps: First we normalize the batch data on a z parameter to Mu=0, sigma^2=1 Then we change z via the ...
1
vote
3answers
90 views

is Batch Norm a little bit stochastic by default?

Using full-batch gradient descent, stacking 100 layers and using alpha 0.0001 results in steadily decreasing error. However, after I implemented Batch Norm, the same scenario results in fluctuations. ...
4
votes
2answers
502 views

Batch normalization variance calculation

In batch normalization the variance calculation during the training phase is done by ($x_i$ are the individual elements in the training batch of size $m$) $\sigma_B^2 = \frac 1m \sum_{i=1}^{m} (x_i -...
8
votes
1answer
456 views

understanding batch normalization

In the paper Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift (here) Before explaining the process of batch normalization the paper tries to explain the ...
4
votes
1answer
3k views

Why is scale parameter on batch normalization not needed on ReLU?

I now read a book titled "Hands-On Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, the author writes the following explanation on batch normalization: Note that by default ...
8
votes
2answers
4k views

Does Batch Normalization make sense for a ReLU activation function?

Batch Normalization is described in this paper as a normalization of the input to an activation function with scale and shift variables $\gamma$ and $\beta$. This paper mainly describes using the ...
30
votes
1answer
10k views

Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?

So, recently there's a Layer Normalization paper. There's also an implementation of it on Keras. But I remember there are papers titled Recurrent Batch Normalization (Cooijmans, 2016) and Batch ...
2
votes
1answer
548 views

Implementing Batch normalisation in Neural network

I have implemented my own mini neural network program1. Currently, it does not have batch updates, it only updates the parameters by simple backpropagation using SGD after each forward pass. I was ...