I am a machine learning researcher who up until this point has primarily worked on Computer Vision problems. However, I have an idea for an NLP technique involving a novel Transformer architecture, and I’d like to explore it.
What’s a good progression of datasets/models to explore? The technique I have in mind is pretty general and should apply to any decoder-only architecture. If it were, say, an image classification problem I might start with ResNet on an MNIST variant or CFAR, then move on to ImageNet. What's the NLP equivalent of that?
Unless there's a better way, I’d like to start by training from scratch on something small and comparing to a vanilla transformer. If things work I’ll want to try my ideas on more state-of-the art models and datasets, probably through fine-tuning. I don’t have a lot of resources, though I do plan to ultimately work up to a GPT-2 fine-tuning if I’m feeling confident.
Thank you in advance!