1
$\begingroup$

This is the project on Neural Machine Translation on the English/Irish language pair. I have been spending the past month or so trying to train a good baseline to do 'experimentation' on. I have a corpus of ~850k sentences (unfortunately Irish is very limiting). When I trained it and evaluated it with BLEU, I got a score of 65.02, which is obviously absurdly incorrect. These were my Fairseq-train settings:

!CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin-full_corp/MayNMT \
  --lr 5e-4 --lr-scheduler inverse_sqrt --optimizer adam\
  --clip-norm 0.1 --dropout 0.2 --max-tokens 4096 \
  --arch transformer --save-dir checkpoints/full-tran

I know not everyone uses Fairseq in NLP, but I hope the arguements are self-explanatory.

I deduplicated the dataset (converted to a Python set() which only takes unique entries) so I don't think the issue is the dev/valid and test sets contain duplicate entries, but I'm not sure what else causes this. Some suggest overfitting may be a cause, but I feel that would only affect the BLEU if the dev set shared training entries. I've tried to find the problem myself, but there aren't many places that cover NMT, let alone BLEU.

$\endgroup$
1
$\begingroup$

According to recent publications, it is not impossible to get BLEU scores as high as yours for English→Irish. Nevertheless, without any other knowledge, they certainly seem too high.

From the command line arguments, there does not seem to be any evident problem.

The most probable explanation is, as you already pointed out, a data leakage between validation/test and training. Note that, while you removed exact duplicates, you may be getting partial matches that go unnoticed. You may want to look into different similarity metrics. The most straightforward is the Jaccard Similarity.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, I'll certainly read up on them. When doing this, should I do it at a document level (my corpus contains about 5/6 different datasets) or sentence level. Or am I getting this wrong altogether (from my understanding from your link, it's comparing short sentences but I can see how it could be scaled up to a document level). $\endgroup$ – Justin Cunningham Jun 16 at 17:18
  • $\begingroup$ I think you should compute the similarity between each validation/test sentence and each training sentence. Given the combinatorial explosion of this, you could reduce the number of comparisons by taking sentence embeddings (e.g. with LASER) and then computing only the Jaccard similarity with the K most similar sentences from the training data. The Faiss library from Facebook may be useful to store sentence embeddings and search the most similar ones. $\endgroup$ – ncasas Jun 16 at 17:58
  • $\begingroup$ And finally, just to clarify, sentences that are deemed too 'similar', can I just put them back into the training set or must they be removed completely? (Sorry if these seem like simple questions, I've learned about NMT online through papers and examples) $\endgroup$ – Justin Cunningham Jun 16 at 18:27
  • $\begingroup$ You should not have similar sentences in the training and validation/test data, so you should remove the similar sentences and leave them only in one set, either training, validation or test. $\endgroup$ – ncasas Jun 16 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.