2
$\begingroup$

when training a VAE, typically one samples from the latent distribution using the reparametrization trick using a fairly large minibatch size (>100) in the decoder/generator half of the VAE. I'm assuming this minibatch size allows the network to 'smooth' out the error and allows us to avoid having to repeatedly sample from the latent space.

However, I'm interested in online scenarios where you are training the VAE on streaming data as it arrives, so the batch size would be 1. In this case, it can take the VAE a long time to converge because the error is highly volatile.

Is there any way to avoid this issue in practice? I am unsure what will happen if I have to repeatedly sample from the latent distribution and then take the mean of those samples (or something) - aside from obvious performance concerns. The other alternative is to wait for enough samples to arrive that I can train them in a larger batch, but even in this case I wouldn't be able to wait for 100+ samples to arrive.

Improve this question
$\endgroup$
2
  • $\begingroup$ The fact that pieces of data arrive one by one does not define the minibatch size. You can have a buffer of size N (with either FIFO or with any other eviction policy that suits the statistical properties needed) that and sample minibatches of size M out it every time you want to update the autoencoder. Depending on the buffer eviction policy, this may also help with local autocorrelation. $\endgroup$
    – noe
    Sep 21, 2016 at 17:43
  • $\begingroup$ Hi ncasas, I have implemented your suggestion and I am satisfied with the results - it behaves as I would expect and doesn't exhibit any of the concerning convergence issues. I have to determine an policy for sampling from the buffer which works the best, but that is something I'll have to experiment with. If you post your comment as an answer, I will flag it as acecpted. $\endgroup$
    – NMR
    Sep 22, 2016 at 12:05

1 Answer 1

Reset to default
1
$\begingroup$

The fact that pieces of data arrive one by one does not define the minibatch size. You can have a buffer of size N (with either FIFO or with any other eviction policy that suits the statistical properties needed) and sample minibatches of size M out it every time you want to update the autoencoder.

Depending on the buffer eviction policy and the sampling strategy, this may also help to avoid local autocorrelation.

Note: this answer was originally a comment to the OP's question.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.