Skip to main content

Questions tagged [mini-batch-gradient-descent]

Is a variation of the gradient descent algorithm that splits the training dataset into small batches that are used to calculate model error and update model coefficients. Implementations may choose to sum the gradient over the mini-batch which further reduces the variance of the gradient. The point of using mini-batch is that you are able to update your weights more than once each epoch, so your model gets better. Mini-batch is considered more efficient.

Filter by
Sorted by
Tagged with
23 votes
2 answers
18k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: <...
Kari's user avatar
  • 2,736
14 votes
2 answers
4k views

Why averaging the gradient works in Gradient Descent?

In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally ...
Kari's user avatar
  • 2,736
8 votes
2 answers
1k views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
lo tolmencre's user avatar
8 votes
1 answer
5k views

sklearn: SGDClassifier yields lower accuracy than LogisticRegression

I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement ...
PrestonH's user avatar
  • 201
6 votes
1 answer
9k views

Does small batch size improve the model?

I'm training an LSTM with Keras. I've noticed that the smaller the batch size, the more the loss decreases during periods: so ...
pairon's user avatar
  • 405
6 votes
1 answer
3k views

Changing the batch size during training

The choice of batch size is in some sense the measure of stochasticity : On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact ...
spiridon_the_sun_rotator's user avatar
6 votes
2 answers
4k views

Latent loss in variational autoencoder drowns generative loss

I'm trying to run a variational auto-encoder on the CIFAR-10 dataset, for which I've put together a simple network in TensorFlow with 4 layers in the encoder and decoder each, an encoded vector size ...
Ali250's user avatar
  • 213
6 votes
1 answer
492 views

How backpropagation through gradient descent represents the error after each forward pass

In Neural NEtwork Multilayer Perceptron, I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while ...
Katherine's user avatar
5 votes
1 answer
3k views

Train loss vs validation loss

I have a few basic questions about tracking losses during training. If I am using mini-batch training, should I validate after each batch update or after I have seen the entire dataset? What should ...
pg2455's user avatar
  • 213
4 votes
2 answers
2k views

In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence?

I want to train an LSTM model with variable length inputs. Specifically I want to use as little padding as possible while still using minibatches. As far as I understand each batch requires a fixed ...
Tonnz's user avatar
  • 63
3 votes
1 answer
3k views

how does minibatch for LSTM look like?

Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our ...
Kari's user avatar
  • 2,736
3 votes
1 answer
495 views

training model on random samples from a large dataset

I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my ...
rawwar's user avatar
  • 861
3 votes
1 answer
263 views

Will stochastic gradient descent converge for multivariate linear regression

I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1). My guess is yes, based on ...
qxzsilver's user avatar
  • 133
3 votes
1 answer
748 views

Plotting Gradient Descent in 3d - Contour Plots

I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
m2rik's user avatar
  • 321
2 votes
1 answer
2k views

mini batch vs. batch gradient descent

In batch gradient descent, it is said that one iteration of gradient descent update takes the processing of whole entire dataset, which I believe makes an epoch.On the other hand, in mini batch ...
Arwen's user avatar
  • 43
2 votes
3 answers
1k views

How much of a problem is each member of a batch having the same label?

I have a batch size of 128 and a total data size of around 10 million, and I am classifying between 4 different label values. How much of a problem is it if each batch only contains data with one ...
Omroth's user avatar
  • 145
2 votes
0 answers
209 views

Vowpal Wabbit Online Normalization -- Possible to parallelize?

Vowpal Wabbit (VW) uses online normalization as explained here [1]. When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch. Is it possible ...
JC1's user avatar
  • 121
1 vote
1 answer
42 views

How to comment on goodness of loss functions?

I have two loss functions $\mathcal{L}_1$ and $\mathcal{L}_2$ to train my model. The model is predominantly a classification model. Both $\mathcal{L}_1$ and $\mathcal{L}_2$ takes are two variants of ...
Aleph's user avatar
  • 123
1 vote
1 answer
202 views

What is the difference between different batch_sizes in Keras Sequential models?

I am interested to know, what happens when I choose batch_size=1 or batch_size=1000 or any other numbers in ...
user3486308's user avatar
  • 1,290
1 vote
1 answer
2k views

Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?

I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent. At least I understand that one iteration of mini-batch gradient descent should ...
Blaszard's user avatar
  • 911
1 vote
2 answers
316 views

Does Minibatch reduce drawback of SGD?

Many expert said "Batch has more local optimal possibility than SGD". But, I don't know the reason.. How SGD could avoid local optimal better than Batch? (Some people tell me about over shooting as ...
douner's user avatar
  • 125
1 vote
1 answer
872 views

Which batch size to use when Batch Normalization?

I want to train a CNN in Keras (optimizer Adam) and by using batch normalization after every ConvLayer and before every activation layer. So far I mostly see examples in which training is carried out ...
Code Now's user avatar
  • 403
1 vote
1 answer
2k views

Powers of 2 for batch_size in model fit in deep learning

I am currently reading Deep Learning with Python by Francois Chollet, the author of Keras, and in one of his definitions for Mini-batch, he explains that the power of 2 for the ...
aspiring1's user avatar
  • 377
1 vote
1 answer
963 views

splitting of training examples into the mini batch: what to do with the rest tiny mini-batch?

Lets assume I have 103 training examples. I want a mini-batch to be of the size 16. That means that there will be 6 mini-batches of the size 16 and one mini-batch of the size 7. In the tensor flow ...
Alina's user avatar
  • 143
1 vote
2 answers
625 views

Setting batch size: training requires twice as much memory as validating

I am using Keras with a Tensorflow backend to train an Image Classification model on a GPU. I have read somewhere that training uses roughly twice (both forward and back props) the GPU memory of ...
Peter Lawrence's user avatar
1 vote
1 answer
49 views

Point of dropping weights in mini batch for purpose of regularization

I have been reading "drop" is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this ...
Miska's user avatar
  • 11
1 vote
1 answer
108 views

Batch Normalization will disrupt multi-threading?

Question: In a Feed Forward network, assume we have a Mini-batch of 64 examples. Our layer $l$ contains 20 neurons Because every neuron in layer will require Mean and Variance from the whole ...
Kari's user avatar
  • 2,736
1 vote
0 answers
19 views

Epochs for new batch when online training?

I am online training a RNN with fixed batch size k on a time series. Initially I train my model with n batches and a number of e epochs. When a new batch n+1 is available, I would like to update the ...
Marx's user avatar
  • 11
1 vote
0 answers
105 views

Relation between batch size, number of steps, and learning rate

Taking alphazero training setup as a reference: 700k total steps batch-size of 4096 initial LR of 0.2 What would be an equivalent setup for a batch-size of 1024? ...
danny's user avatar
  • 111
1 vote
1 answer
502 views

Why does neural network need loss as scalar?

I have a loss function that's a weighted cross entropy loss for binary classification ...
rakeshKM's user avatar
1 vote
2 answers
447 views

Should mini-batches contain an even mix of classes or can this be random?

I'm creating mini-batches to put into a CNN. Is it best to try and get an even mix of classes into each mini-batch (Scenario 1), or can this/should this be a random assortment of my classes (Scenario ...
TheyTakingTheHobbitsToIsengard's user avatar
1 vote
0 answers
346 views

Tuning Batch size and Learning rate in neural net

The following MCQ question is provided in "Exam Readiness: AWS Certified Machine Learning - Specialty" document. The correct answer has been marked in the document but I am not able to ...
Suvra Dutta's user avatar
1 vote
0 answers
12 views

why would you mask out padded activations from the training loss?

I've followed taming-lstm for training a LSTM model on a NLP task in batches with various sentence lengths. One of his main points is: Trick 3: Mask out network outputs we don’t want to consider in ...
ihadanny's user avatar
  • 1,357
1 vote
0 answers
218 views

How to implement large-scale Poisson Regression in Python

I am trying to implement a Poisson Regression in Python to predict rates. I am dealing with a ton of data (too much to store in a DataFrame), which means that using the standard statsmodels.api GLM ...
Michael Petro's user avatar
1 vote
1 answer
76 views

Displaying network error as a single value

I've been writing a neural network from scratch. I've completed the feedforward, backpropagation, and mini-batch gradient descent methods, so I can train the network. Other neural networks I've worked ...
Shan S's user avatar
  • 148
1 vote
0 answers
217 views

Tensorflow - Manually decay Adam optimizer

I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the ...
SuperCodeBrah's user avatar
1 vote
1 answer
188 views

SGD vs SGD in mini batches

So I recently finished a mini batches algorithm for a library in building in java(artificial neural network lib). I then followed to train my network for an XOR problem in mini batches size of 2 or 3, ...
Itay Bachar's user avatar
1 vote
0 answers
102 views

Mini-batches with sequential data

I am a little bit confused. When using mini-batches, it is a good idea to shuffle. This will not work if the training examples are dependent on each other, e.g. 5 minute voltage measurement data, ...
Ben Groene's user avatar
1 vote
1 answer
266 views

Online vs minibatch training for speed

If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 ...
StatsSorceress's user avatar
0 votes
2 answers
2k views

Why Mini batch gradient descent is faster than gradient descent?

According to me: Mini Batch Gradient Descent : 1.It takes a specified batch number say 32. 2.Evaluate loss on 32 examples. 3.Update weights. 4.Repeat until every example is complete. 5.Repeat till a ...
Shiv's user avatar
  • 689
0 votes
1 answer
326 views

With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

In a blog I read this: With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch. blog. Now I am confused with the whole ...
Shiv's user avatar
  • 689
0 votes
1 answer
421 views

In Mini Batch Gradient Descent what happens to remaining examples

Suppose my dataset has 1000 samples (X=1000) . I choose batch size of 32. As 1000 is not perfectly divisible by 32 , remainder is 8. My question is what happens to the last 8 examples. Are they ...
Shiv's user avatar
  • 689
0 votes
1 answer
39 views

Why does forecasting with an LSTM yield better results with shuffling?

I first partition the timeseries data into train, validation, and test splits, without performing any shuffling. Each row is a window of ordered samples, so my training data might be shaped ...
MuhammedYunus's user avatar
0 votes
1 answer
253 views

How do i get the loss function graph?

I used Mini-batch gradient descent to train the model, but i am unable to get the proper loss graph. The loss graph is always showed as a straight line. I know there is something wrong but would ...
xstrx's user avatar
  • 1
0 votes
1 answer
672 views

Why is my LSTM is working best with batch size of 2 and no hidden layers?

I am building an LSTM for price prediction using Keras. I am using Bayesian optimization to find the right hyperparameters. With ...
user134132523's user avatar
0 votes
1 answer
196 views

PyTorch MultiLayer Perceptron Classification Size of Features vs Labels Wrong

I am getting the following error: ...
user91925's user avatar
0 votes
1 answer
298 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
Leevo's user avatar
  • 6,265
0 votes
1 answer
88 views

How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
yamini goel's user avatar
0 votes
0 answers
30 views

Full batch vs mini-batch gradient descent

Help me solve this problem! A training dataset has 10 classes and each class has 1000 samples. The samples in each class are redundant. If you apply full batch gradient descent, it takes 1000 updates ...
Prabal's user avatar
  • 1
0 votes
0 answers
50 views

In MLP, multiple classes, using batches, For update weights, Would I have to calculate the accumulated error(of all samples) of each output neuron?

In Multilayer Perceptron neural networks, I know that there are two types of training: online training, and batch training, which consists of dividing the samples and updating the weights using the ...
will The J's user avatar