Questions tagged [mini-batch-gradient-descent]

Filter by
Sorted by
Tagged with
5
votes
2answers
67 views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
0
votes
1answer
22 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
0
votes
0answers
14 views

Is there any literature on how the batch-size affects the estimation of parameters of the output distribution in Variational Autoencoder?

I am working on data imputation task. I am using variational autoencoder to estimate the values of missing data. The data has real, categorical and textual features. I assume the output distribution ...
0
votes
1answer
35 views

How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
0
votes
1answer
51 views

SGD vs SGD in mini batches

So I recently finished a mini batches algorithm for a library in building in java(artificial neural network lib). I then followed to train my network for an XOR problem in mini batches size of 2 or 3, ...
0
votes
1answer
21 views

Can we use decreasing step size to replace mini-batch in SGD?

As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample ...
1
vote
1answer
35 views

What is the difference between different batch_sizes in Keras Sequential models?

I am interested to know, what happens when I choose batch_size=1 or batch_size=1000 or any other numbers in ...
1
vote
0answers
49 views

Mini-batches with sequential data

I am a little bit confused. When using mini-batches, it is a good idea to shuffle. This will not work if the training examples are dependent on each other, e.g. 5 minute voltage measurement data, ...
1
vote
1answer
204 views

Powers of 2 for batch_size in model fit in deep learning

I am currently reading Deep Learning with Python by Francois Chollet, the author of Keras, and in one of his definitions for Mini-batch, he explains that the power of 2 for the ...
3
votes
1answer
131 views

training model on random samples from a large dataset

I have a huge data set(More than 1 million data points).My dataset is text. i am doing NER on it to identify few entities. if i randomly choose 100 data points from the total data set and train my ...
0
votes
1answer
194 views

splitting of training examples into the mini batch: what to do with the rest tiny mini-batch?

Lets assume I have 103 training examples. I want a mini-batch to be of the size 16. That means that there will be 6 mini-batches of the size 16 and one mini-batch of the size 7. In the tensor flow ...
1
vote
2answers
249 views

Setting batch size: training requires twice as much memory as validating

I am using Keras with a Tensorflow backend to train an Image Classification model on a GPU. I have read somewhere that training uses roughly twice (both forward and back props) the GPU memory of ...
4
votes
2answers
1k views

Latent loss in variational autoencoder drowns generative loss

I'm trying to run a variational auto-encoder on the CIFAR-10 dataset, for which I've put together a simple network in TensorFlow with 4 layers in the encoder and decoder each, an encoded vector size ...
3
votes
2answers
461 views

Why averaging the gradient works in Gradient Descent?

In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally ...
1
vote
0answers
43 views

Test data also being processed in batches

...
3
votes
1answer
1k views

Train loss vs validation loss

I have a few basic questions about tracking losses during training. If I am using mini-batch training, should I validate after each batch update or after I have seen the entire dataset? What should ...
1
vote
1answer
23 views

Point of dropping weights in mini batch for purpose of regularization

I have been reading "drop" is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this ...
1
vote
0answers
163 views

Online vs minibatch training for speed

If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 ...
2
votes
0answers
122 views

Vowpal Wabbit Online Normalization — Possible to parallelize?

Vowpal Wabbit (VW) uses online normalization as explained here [1]. When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch. Is it possible ...
14
votes
2answers
4k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: ...
1
vote
1answer
69 views

Batch Normalization will disrupt multi-threading?

Question: In a Feed Forward network, assume we have a Mini-batch of 64 examples. Our layer $l$ contains 20 neurons Because every neuron in layer will require Mean and Variance from the whole ...
1
vote
1answer
1k views

how does minibatch for LSTM look like?

Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our ...
5
votes
1answer
187 views

How backpropagation through gradient descent represents the error after each forward pass

In Neural NEtwork Multilayer Perceptron, I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while ...
5
votes
1answer
3k views

sklearn: SGDClassifier yields lower accuracy than LogisticRegression

I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement ...
1
vote
1answer
877 views

Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?

I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent. At least I understand that one iteration of mini-batch gradient descent should ...