1
$\begingroup$

As reference to Why should the data be shuffled for machine learning tasks I was curious if this is the case for neural-machine-translation.

I would like to analyze why I think this is the case for the 3 components (train, test, val) parts where the data (as a reference dataset think of the europarl-dataset) is split:

  • for the training part, when doing mini-batch gradient descent it is better that for each update we collect data that are closer semantically, so that the learning gets focused each time on a certain direction. For example is better to have a lot of topics for let's say coffee together for a single update, than topics about drugs or coffee, dogs etc in a single update

  • for the test part, we would like to see how the model behaves in a big text of continuous sentences, as the goal of translation is that of documents. The same should hold for validation

I would like your view on that if you have any empirical or theoretical insight of why this is not the case for neural machine translation.

$\endgroup$
0
$\begingroup$

Yes, in NMT data is always shuffled.

For training, if each batch contains the same "type of content" (e.g. domain, register), then your model will be biased toward the type of content of the last batches. You don't want that. Apart from that, in NMT data is normally bucketed by length to avoid wasting batch space with padding.

For the test data, you would like to have data that follows a similar distribution to the one to be used in production. If the test is biased to only one domain, then you won't be able to assess correctly the expected performance of the model.

Document-level translation has little to do with the shuffling of the data. In document-level MT you normally have context sentences that are either ingested with an auxiliary encoder or as part of a single, longer sentence. This may reduce the number of sentences you can fit in the batch, but each sentence (+context) is independent from the rest.

$\endgroup$
4
  • $\begingroup$ "In document-level MT you normally have context sentences that are either ingested with an auxiliary encoder" <- Can you maybe pinpoint me to an implementation concerning that or to a paper? $\endgroup$
    – user86834
    Dec 15 '19 at 17:11
  • $\begingroup$ Marian supports long sentences (i.e. whole documents) via gradient checkpointing and auxiliary encoders (normally used for automatic post-editing). $\endgroup$
    – noe
    Dec 16 '19 at 0:04
  • $\begingroup$ Nice! Can you may send me a link, or some command line options? $\endgroup$
    – user86834
    Dec 16 '19 at 12:11
  • $\begingroup$ You can check the marian examples page for plenty of examples. Search for post-editing in that page and you will find the dual encoder thing. The gradient checkpointing is a new addition and I don't know of any documentation yet. $\endgroup$
    – noe
    Dec 17 '19 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.