Skip to main content

Questions tagged [mini-batch-gradient-descent]

Is a variation of the gradient descent algorithm that splits the training dataset into small batches that are used to calculate model error and update model coefficients. Implementations may choose to sum the gradient over the mini-batch which further reduces the variance of the gradient. The point of using mini-batch is that you are able to update your weights more than once each epoch, so your model gets better. Mini-batch is considered more efficient.

Filter by
Sorted by
Tagged with
1 vote
1 answer
266 views

Online vs minibatch training for speed

If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 ...
StatsSorceress's user avatar
2 votes
0 answers
209 views

Vowpal Wabbit Online Normalization -- Possible to parallelize?

Vowpal Wabbit (VW) uses online normalization as explained here [1]. When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch. Is it possible ...
JC1's user avatar
  • 121
23 votes
2 answers
18k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: <...
Kari's user avatar
  • 2,736
1 vote
1 answer
108 views

Batch Normalization will disrupt multi-threading?

Question: In a Feed Forward network, assume we have a Mini-batch of 64 examples. Our layer $l$ contains 20 neurons Because every neuron in layer will require Mean and Variance from the whole ...
Kari's user avatar
  • 2,736
3 votes
1 answer
3k views

how does minibatch for LSTM look like?

Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our ...
Kari's user avatar
  • 2,736
6 votes
1 answer
492 views

How backpropagation through gradient descent represents the error after each forward pass

In Neural NEtwork Multilayer Perceptron, I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while ...
Katherine's user avatar
8 votes
1 answer
5k views

sklearn: SGDClassifier yields lower accuracy than LogisticRegression

I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement ...
PrestonH's user avatar
  • 201
1 vote
1 answer
2k views

Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?

I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent. At least I understand that one iteration of mini-batch gradient descent should ...
Blaszard's user avatar
  • 911

1
2