Questions tagged [mini-batch-gradient-descent]
Is a variation of the gradient descent algorithm that splits the training dataset into small batches that are used to calculate model error and update model coefficients. Implementations may choose to sum the gradient over the mini-batch which further reduces the variance of the gradient. The point of using mini-batch is that you are able to update your weights more than once each epoch, so your model gets better. Mini-batch is considered more efficient.
59
questions
0
votes
0
answers
8
views
Polynomial Regression coefficient extraction after data normalisation for Mini-Batch SGD
I've written python function that uses a stochastic mini-batch algorithm to compute the optimal polynomial coefficients for a given degree $m$, however this involved normalising the data where
$$
x' = ...
- 101
1
vote
0
answers
14
views
Epochs for new batch when online training?
I am online training a RNN with fixed batch size k on a time series. Initially I train my model with n batches and a number of e epochs. When a new batch n+1 is available, I would like to update the ...
- 11
0
votes
0
answers
21
views
Why does merging all loss in a batch make sense?
I built a vanilla GRU net to resolve a binary classification problem, and train it with a batch size greater than 1.
Let's say my NN model got a correct result with sample-0, and got a incorrect ...
- 101
0
votes
0
answers
10
views
When do we train neural networks in full batch?
It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that ...
- 131
0
votes
1
answer
132
views
Gradient descent vs stochastic gradient descent vs mini-batch gradient descent with respect to working step/example
I am trying to understand the working of gradient descent, stochastic gradient descent and mini-batch gradient descent.
In case of gradient descent, gradient is computed on the entire dataset at each ...
- 187
1
vote
0
answers
72
views
Relation between batch size, number of steps, and learning rate
Taking alphazero training setup as a reference:
700k total steps
batch-size of 4096
initial LR of 0.2
What would be an equivalent setup for a batch-size of 1024? ...
- 111
0
votes
0
answers
16
views
Why would an affine forward layer take unflattened inputs?
I'm doing Stanford's CS 231n on my own.
I'm tasked with implementing the forward function for an affine forward layer.
Here's the doc comment:
...
- 115
0
votes
0
answers
26
views
What will happen if we train a model on a dataset sorted by class
Suppose we have a dataset of two classes (0 and 1) divided into over 12k mini-batches where the first half of the dataset (over 6k mini-batches) belong to class 0, and the other half belongs to class ...
0
votes
0
answers
43
views
What is the technical name for a "batch element" in ML?
What is the technical name for a "batch element" in machine learning?
Given a batch of data (size: batchSize*numberOfFeatures), what is the technical name ...
- 121
0
votes
0
answers
25
views
How are session-parallel mini-batches used for training RNNs for session-based recommender tasks?
I am reading this paper on session-based recommenders with RNNs: https://arxiv.org/abs/1511.06939. During the training phase, the authors apply what they call "session-parallel mini-batches,"...
0
votes
1
answer
218
views
Why does neural network need loss as scalar?
I have a loss function that's a weighted cross entropy loss for binary classification
...
- 1
1
vote
2
answers
142
views
Should mini-batches contain an even mix of classes or can this be random?
I'm creating mini-batches to put into a CNN. Is it best to try and get an even mix of classes into each mini-batch (Scenario 1), or can this/should this be a random assortment of my classes (Scenario ...
1
vote
0
answers
241
views
Tuning Batch size and Learning rate in neural net
The following MCQ question is provided in "Exam Readiness: AWS Certified Machine Learning - Specialty" document. The correct answer has been marked in the document but I am not able to ...
- 21
0
votes
0
answers
42
views
Can the 'Rainbow Algorithm' be scaled up and sped up?
What's the proper way to train the algorithm with bigger batches or otherwise speed it up?
The 'Rainbow Algorithm' is a Deep Q, Reinforcement Learning algorithm with two neural networks that I would ...
- 167
0
votes
1
answer
161
views
How do i get the loss function graph?
I used Mini-batch gradient descent to train the model, but i am unable to get the proper loss graph. The loss graph is always showed as a straight line. I know there is something wrong but would ...
- 1
1
vote
0
answers
10
views
why would you mask out padded activations from the training loss?
I've followed taming-lstm for training a LSTM model on a NLP task in batches with various sentence lengths. One of his main points is:
Trick 3: Mask out network outputs we don’t want to consider in ...
- 1,337
0
votes
1
answer
37
views
Why are mini-batches degrading my conv net MNIST classifier?
I have made a convolutional neural network from scratch in python to classify the MNIST handwritten digits (centralized). It is composed of a single convolutional network with 8 3x3 kernels, a 2x2 ...
6
votes
1
answer
2k
views
Changing the batch size during training
The choice of batch size is in some sense the measure of stochasticity :
On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact ...
0
votes
1
answer
23
views
Neural Network Optimization steps order
I have a very basic question on the optimization algotithm, when I'm adjusting weights and biases in a NN, should I:
Forward propagate and backpropagate to calculate gradient descent (DC) for each ...
4
votes
2
answers
1k
views
In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence?
I want to train an LSTM model with variable length inputs. Specifically I want to use as little padding as possible while still using minibatches.
As far as I understand each batch requires a fixed ...
- 63
1
vote
0
answers
158
views
How to implement large-scale Poisson Regression in Python
I am trying to implement a Poisson Regression in Python to predict rates. I am dealing with a ton of data (too much to store in a DataFrame), which means that using the standard statsmodels.api GLM ...
0
votes
2
answers
1k
views
Why Mini batch gradient descent is faster than gradient descent?
According to me:
Mini Batch Gradient Descent :
1.It takes a specified batch number say 32.
2.Evaluate loss on 32 examples.
3.Update weights.
4.Repeat until every example is complete.
5.Repeat till a ...
- 669
0
votes
1
answer
173
views
With Stochastic Gradient Descent why we dont compute exact derivative of loss function?
In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
blog.
Now I am confused with the whole ...
- 669
0
votes
1
answer
243
views
In Mini Batch Gradient Descent what happens to remaining examples
Suppose my dataset has 1000 samples (X=1000) . I choose batch size of 32.
As 1000 is not perfectly divisible by 32 , remainder is 8.
My question is what happens to the last 8 examples. Are they ...
- 669
3
votes
1
answer
213
views
Will stochastic gradient descent converge for multivariate linear regression
I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1).
My guess is yes, based on ...
- 133
2
votes
3
answers
704
views
How much of a problem is each member of a batch having the same label?
I have a batch size of 128 and a total data size of around 10 million, and I am classifying between 4 different label values.
How much of a problem is it if each batch only contains data with one ...
- 145
0
votes
1
answer
501
views
Why is my LSTM is working best with batch size of 2 and no hidden layers?
I am building an LSTM for price prediction using Keras. I am using Bayesian optimization to find the right hyperparameters. With ...
- 149
2
votes
1
answer
968
views
mini batch vs. batch gradient descent
In batch gradient descent, it is said that one iteration of gradient descent update takes the processing of whole entire dataset, which I believe makes an epoch.On the other hand, in mini batch ...
- 43
0
votes
1
answer
1k
views
Mini Batch Gradient Descent shuffling
My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?
1
vote
1
answer
50
views
Displaying network error as a single value
I've been writing a neural network from scratch. I've completed the feedforward, backpropagation, and mini-batch gradient descent methods, so I can train the network. Other neural networks I've worked ...
- 148
6
votes
1
answer
6k
views
Does small batch size improve the model?
I'm training an LSTM with Keras.
I've noticed that the smaller the batch size, the more the loss decreases during periods: so ...
- 375
1
vote
0
answers
202
views
Tensorflow - Manually decay Adam optimizer
I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the ...
- 181
0
votes
1
answer
166
views
PyTorch MultiLayer Perceptron Classification Size of Features vs Labels Wrong
I am getting the following error:
...
3
votes
1
answer
637
views
Plotting Gradient Descent in 3d - Contour Plots
I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
- 321
1
vote
2
answers
265
views
Does Minibatch reduce drawback of SGD?
Many expert said "Batch has more local optimal possibility than SGD". But, I don't know the reason..
How SGD could avoid local optimal better than Batch?
(Some people tell me about over shooting as ...
- 115
1
vote
1
answer
762
views
Which batch size to use when Batch Normalization?
I want to train a CNN in Keras (optimizer Adam) and by using batch normalization after every ConvLayer and before every activation layer.
So far I mostly see examples in which training is carried out ...
- 393
8
votes
2
answers
1k
views
Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?
I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
- 235
0
votes
1
answer
270
views
Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?
I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
- 5,897
0
votes
1
answer
77
views
How to find learning rate decay?
Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
- 701
1
vote
1
answer
150
views
SGD vs SGD in mini batches
So I recently finished a mini batches algorithm for a library in building in java(artificial neural network lib). I then followed to train my network for an XOR problem in mini batches size of 2 or 3, ...
- 13
0
votes
2
answers
143
views
Can we use decreasing step size to replace mini-batch in SGD?
As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample ...
- 101
1
vote
1
answer
163
views
What is the difference between different batch_sizes in Keras Sequential models?
I am interested to know, what happens when I choose batch_size=1 or batch_size=1000 or any other numbers in ...
- 1,182
1
vote
0
answers
96
views
Mini-batches with sequential data
I am a little bit confused.
When using mini-batches, it is a good idea to shuffle. This will not work if the training examples are dependent on each other, e.g. 5 minute voltage measurement data, ...
- 51
1
vote
1
answer
1k
views
Powers of 2 for batch_size in model fit in deep learning
I am currently reading Deep Learning with Python by Francois Chollet, the author of Keras, and in one of his definitions for Mini-batch, he explains that the power of 2 for the ...
- 357
3
votes
1
answer
431
views
training model on random samples from a large dataset
I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my ...
- 811
1
vote
1
answer
876
views
splitting of training examples into the mini batch: what to do with the rest tiny mini-batch?
Lets assume I have 103 training examples. I want a mini-batch to be of the size 16. That means that there will be 6 mini-batches of the size 16 and one mini-batch of the size 7.
In the tensor flow ...
- 143
1
vote
2
answers
571
views
Setting batch size: training requires twice as much memory as validating
I am using Keras with a Tensorflow backend to train an Image Classification model on a GPU. I have read somewhere that training uses roughly twice (both forward and back props) the GPU memory of ...
- 111
5
votes
2
answers
3k
views
Latent loss in variational autoencoder drowns generative loss
I'm trying to run a variational auto-encoder on the CIFAR-10 dataset, for which I've put together a simple network in TensorFlow with 4 layers in the encoder and decoder each, an encoded vector size ...
- 193
13
votes
2
answers
3k
views
Why averaging the gradient works in Gradient Descent?
In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally ...
- 2,676
5
votes
1
answer
3k
views
Train loss vs validation loss
I have a few basic questions about tracking losses during training.
If I am using mini-batch training, should I validate after each
batch update or after I have seen the entire dataset?
What should ...
- 213