I don't know if this is what you are looking for, but Andrej Karpathy has a good blog article about his method for training networks in general: A Recipe for Training Neural Networks
I will put the bullet points of his recipe here, but there is way more practical advice in the actual blog.
The recipe
- Become one with the data
- Set up the end-to-end training/evaluation skeleton + get dumb baselines
- Overfit
- Regularize
- Tune
- Squeeze out the juice
I think the advice that best fit your situation is from step 2:
At this stage it is best to pick some simple model that you couldn’t
possibly have screwed up somehow - e.g. a linear classifier, or a very
tiny ConvNet. We’ll want to train it, visualize the losses, any other
metrics (e.g. accuracy), model predictions, and perform a series of
ablation experiments with explicit hypotheses along the way.
You usually want to start with something simple that will not allow you to have issues with overfitting like you have. Then you will progressively work your way into a more complex model that can overfit in step 3.