I'm working on making a transformer from scratch as described in the "Attention is All You Need" Paper. When training my model, my cross-entropy loss is always very low at the start. For example, my first epoch saw a loss of 8.16e-5. Is it normal for Cross Entropy loss to be this low right from the start?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.