I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For example, my first epoch saw a loss of 8.16e-5. Is it normal for Cross Entropy loss to be this low right from the start?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Anybody can ask a question
Anybody can answer
The best answers are voted up and rise to the top
