A good set of datasets/models for testing an NLP technique

Question

I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, and I’d like to explore it.

What’s a good progression of datasets/models to explore? The technique I have in mind is pretty general and should apply to any decoder-only architecture. If it were, say, an image classification problem I might start with ResNet on an MNIST variant or CFAR, then move on to ImageNet. What's the NLP equivalent of that?

Unless there's a better way, I’d like to start by training from scratch on something small and comparing to a vanilla transformer. If things work I’ll want to try my ideas on more state-of-the art models and datasets, probably through fine-tuning. I don’t have a lot of resources, though I do plan to ultimately work up to a GPT-2 fine-tuning if I’m feeling confident.

Thank you in advance!

Answer 1

For datasets:

• To the best of my knowledge, the IMDB dataset is serving as the "Hello World" of NLP, especially for LLM-related matters, and can be used for either binary classification or generative text.
• Another well-known dataset is the BookCorpus which was used to train GPT, BERT, and other LLMs.
• If you're looking to scale up, the Amazon review dataset is also well-known, and there are many existing subsets which are more tractable than the full dataset.

Regarding modeling:

• If you're starting from scratch, you probably want to take a look at Karpathy's minGPT and there are now many "Transformers from Scratch" type posts / videos / tutorials out there
• You want to work with a foundational model to get started (BERT, T5, or GPT) and since you specifically mentioned decoders, would most likely be GPT.
• If you really want to go back to fundamentals, you can take a look at the original Attention is All You Need Paper and code

Hope this is helpful!