3
$\begingroup$

Minibatch is a collection of examples that are fed into the network, (example after example), and back-prop is done after every single example. We then take average of these gradients and update our weights. This completes processing 1 minibatch.

I read these posts

Question part a:

How a minibatch entity would look like for LSTM? Say, I want it to reproduce Shakespeare, letter by letter (30 characters to choose from).

I launch LSTM, let it predict for 200 characters of a poem, then perform back propagation. (hence, my LSTM works with 200 timesteps). Does this mean my minibatch consist of 1 example whose length is 200?

Question part b:

If I wanted to launch 63 other minibatches in parallel, would I just pick 63 extra poems? (Edit: Original answer doesn't mention this explicitly, but we don't train minibatches in parallel. We train on 1 minibatch, but train its examples in parallel)

Question part C:

If I wanted each minibatch to consist of 10 different examples, what would such examples be, and how would they be different from 'what I perceive as a minibatch'?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

I think you need to distinguish between training and execution of the model. During training, you can use batches, which in your case will be different fragments from Shakespeare. So, a batch will be a list of fragments, and the language model will start from the first character on each element of the batch and do the backward and forward pass.

When you execute the model, once it is trained, you would like to see one single example, in which case you can set the batch size to one.

I believe this answers your three questions.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you @Escachator, I need a little clarification, - do I understand correctly: a batch is a list of fragments (in other words a batch is a list of elements), where each fragment is a sequence of characters (one such a fragment could be 200 characters long). Thus, a fragment could represent some distinct sequence of sentences from Shakespere. $\endgroup$
    – Kari
    Commented Dec 28, 2017 at 15:33
  • 1
    $\begingroup$ A batch, in general terms, is a set of samples from the population you want to predict. In a normal predictive model, a batch will be a set of "x" which you use to predict the "y". In a language model, usually you predict the next token (e.g. character) given the previous set of tokens. The way usual language models are programmed, a batch will consist on exactly what you say: a list of lists, where each element of the list will be a distinct fragment. Note that the fragment should be encoded, i.e. you should map each character to a number. $\endgroup$
    – Escachator
    Commented Dec 28, 2017 at 15:40
  • $\begingroup$ A nice tutorial could be: pytorch.org/tutorials/intermediate/… $\endgroup$
    – Escachator
    Commented Dec 28, 2017 at 15:40
  • $\begingroup$ Thanks, so a 'fragment = character' in my concrete example, which should be one-hot encoded $\endgroup$
    – Kari
    Commented Dec 28, 2017 at 16:07
  • 1
    $\begingroup$ When I mean a fragment I meant a sentence or a part of a Shakespeare book. A token would be a character. You can represent a character as a one-hot encoded vector or by an embedding (possibly better). Usually you assign a unique number to a character and then you map it to an embedding. $\endgroup$
    – Escachator
    Commented Dec 28, 2017 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.