I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, and I’d like to explore it.

What’s a good progression of datasets/models to explore? The technique I have in mind is pretty general and should apply to any decoder-only architecture. If it were, say, an image classification problem I might start with ResNet on an MNIST variant or CFAR, then move on to ImageNet. What's the NLP equivalent of that?

Unless there's a better way, I’d like to start by training from scratch on something small and comparing to a vanilla transformer. If things work I’ll want to try my ideas on more state-of-the art models and datasets, probably through fine-tuning. I don’t have a lot of resources, though I do plan to ultimately work up to a GPT-2 fine-tuning if I’m feeling confident.

Thank you in advance!


1 Answer 1


For datasets:

  • To the best of my knowledge, the IMDB dataset is serving as the "Hello World" of NLP, especially for LLM-related matters, and can be used for either binary classification or generative text.
  • Another well-known dataset is the BookCorpus which was used to train GPT, BERT, and other LLMs.
  • If you're looking to scale up, the Amazon review dataset is also well-known, and there are many existing subsets which are more tractable than the full dataset.

Regarding modeling:

  • If you're starting from scratch, you probably want to take a look at Karpathy's minGPT and there are now many "Transformers from Scratch" type posts / videos / tutorials out there
  • You want to work with a foundational model to get started (BERT, T5, or GPT) and since you specifically mentioned decoders, would most likely be GPT.
  • If you really want to go back to fundamentals, you can take a look at the original Attention is All You Need Paper and code

Hope this is helpful!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.