I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time-consuming affair. What are good toy problems to test if a transformer (or alternative thereof) is working at all? I am looking for simple problems that can preferably be synthetically created and can be trained with really small setups (few layers, small embedding sizes, etc.) in a short time. Ideally, these should be problems that play to the strengths of transformers but would be hard to solve for, say, a fully connected feed-forward network. Tasks that can be applied to just an Attention Layer would be useful, too.
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
For MT, I always use the Multi30k dataset, English to German for debugging. It has only 30k sentences which are simple and template-like, with a correctly configured Transformer model, you should get around 30 BLEU points in 2 minutes.
My experience is that toy problems such as copying, capitalization, reversing character order in words are too simple and the model can learn them despite severe bugs that prevent models for real problems from training.
$\begingroup$
$\endgroup$
I would recommend the pre-processed datasets for language translation in the Stanford NLP group: https://nlp.stanford.edu/projects/nmt/
There are three datasets with sizes 0.13Mb, 4.5Mb and 15Mb - something for everybody :)
Translation is something that is very difficult to solve for normal FF-networks and something that transformers dramatically improved due to attention mechanisms.
Also check out the related explanations of those ideas in The Annotated Transformer and the Illustrated Transformer
$\begingroup$
$\endgroup$
Text reversal task is a typical toy problem, not only for Transformers but for seq2seq models in general.
In that task, you take a piece text in whatever language as source, and the words in reverse order as target.
