1
$\begingroup$

Background

I am using scikit-learn's MLPRegressor to learn a model with the following arguments:

hidden_layer_sizes=(16, 16),
learning_rate_init=1e-2,
solver='sgd',
momentum=0.9,
activation='tanh',
learning_rate='constant'

I use partial_fit() to incrementally fit the model. Occasionally I get a FloatingPointError on this line in the MLPRegressor's code:

updates = [self.momentum * velocity - self.learning_rate * grad
                   for velocity, grad in zip(self.velocities, grads)]

>>> FloatingPointError: underflow encountered in multiply

Problem

I am getting an underflow error when the model is calculating how much to update parameters by multiplying learning rate, momentum, velocity, and error gradients.

Does not an underflow mean that the updates are too small? Does it imply the model has reached some local minimum? Should I take this as a hint that:

  • I have exhausted my data and training has ended,
  • I should up my learning rate to keep updates happening
  • Something else?
$\endgroup$
1
$\begingroup$

Underflow is happening because the result of that calculation is a number of smaller absolute value than your computer can actually represent in memory.

Underflow is not directly related to the performance of the model (e.g., learning rate or local minimum). Underflow is the interplay between computer software and hardware.

I suggest rescaling the data. For example, batch normalization often stops underflow errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.