I'm using a Transformer encoder with a binary cross entropy loss for CTR prediction. The training batch loss is at around 0.693 constantly for the beginning several thousand steps (batches). I'm using Noam learning rate schedule and a scaling factor for setting the learning rate used by Adam. The first 3000 steps are the warm up steps in Noam:
lr= factor * ( model_size ** (-0.5) * min(step** (-0.5), step * warmup_steps ** (-1.5)) )
For the learning rate, when setting scaling factor to be 0.5 or 0.6, the period with constant 0.693 is from beginning to 5500th steps. When I use lower lr by setting scaling factor 0.3, the period with constant 0.693 became longer (8000 steps). When I use higher lr by setting scaling factor 1.0, the period with constant 0.693 became shorter (4000 steps).
Note: the above training batch loss curve is smoothed.
Does this mean I should use a high learning rate (e.g., with scale factor being 1.0)? Why there is a period with constant 0.693 loss? is it because the learning rate is too small to make any adjustment to the parameters? If so, after the 3000 warmup steps the learning rate is even smaller, why the loss is decreasing?
