What is the significance of underflow during parameter update using stochastic gradient descent?

Question

Background

I am using scikit-learn's MLPRegressor to learn a model with the following arguments:

hidden_layer_sizes=(16, 16),
learning_rate_init=1e-2,
solver='sgd',
momentum=0.9,
activation='tanh',
learning_rate='constant'

I use partial_fit() to incrementally fit the model. Occasionally I get a FloatingPointError on this line in the MLPRegressor's code:

updates = [self.momentum * velocity - self.learning_rate * grad
                   for velocity, grad in zip(self.velocities, grads)]

>>> FloatingPointError: underflow encountered in multiply

Problem

I am getting an underflow error when the model is calculating how much to update parameters by multiplying learning rate, momentum, velocity, and error gradients.

Does not an underflow mean that the updates are too small? Does it imply the model has reached some local minimum? Should I take this as a hint that:

• I have exhausted my data and training has ended,
• I should up my learning rate to keep updates happening
• Something else?

Answer 1

Underflow is happening because the result of that calculation is a number of smaller absolute value than your computer can actually represent in memory.

Underflow is not directly related to the performance of the model (e.g., learning rate or local minimum). Underflow is the interplay between computer software and hardware.

I suggest rescaling the data. For example, batch normalization often stops underflow errors.