1
$\begingroup$

I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer (or alternative thereof) is working at all? I am looking for simple problems that can preferably be synthetically created and can be trained with really small setups (few layers, small embedding sizes etc.) in a short time. Ideally this should be problems that play to the strengths of transformers but would be hard to solve for, say, a fully connected feed-forward network. Tasks that can be applied to just an Attention Layer would be useful, too.

$\endgroup$
0
$\begingroup$

I would recommend the pre-processed datasets for language translation in the Stanford NLP group: https://nlp.stanford.edu/projects/nmt/

There are three datasets with sizes 0.13Mb, 4.5Mb and 15Mb - something for everybody :)


Translation is something that is very difficult to solve for normal FF-networks and something that transformers dramatically improved due to attention mechanisms.

Also check out the related explanations of those ideas in The Annotated Transformer and the Illustrated Transformer

| improve this answer | |
$\endgroup$
0
$\begingroup$

For MT, I always use the Multi30k dataset, English to German for debugging. It has only 30k sentences which are simple and template-like, with a correctly configured Transformer model, you should get around 30 BLEU points in 2 minutes.

My experience is that toy problems such as copying, capitalization, reversing character order in words are too simple and the model can learn them despite severe bugs that prevent models for real problems from training.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.