1
$\begingroup$

Background

I am using scikit-learn's MLPRegressor to learn a model with the following arguments:

hidden_layer_sizes=(16, 16),
learning_rate_init=1e-2,
solver='sgd',
momentum=0.9,
activation='tanh',
learning_rate='constant'

I use partial_fit() to incrementally fit the model. Occasionally I get a FloatingPointError on this line in the MLPRegressor's code:

updates = [self.momentum * velocity - self.learning_rate * grad
                   for velocity, grad in zip(self.velocities, grads)]

>>> FloatingPointError: underflow encountered in multiply

Problem

I am getting an underflow error when the model is calculating how much to update parameters by multiplying learning rate, momentum, velocity, and error gradients.

Does not an underflow mean that the updates are too small? Does it imply the model has reached some local minimum? Should I take this as a hint that:

  • I have exhausted my data and training has ended,
  • I should up my learning rate to keep updates happening
  • Something else?
$\endgroup$
2
  • 1
    $\begingroup$ try to change your inputs types to np.float64 $\endgroup$ Nov 16, 2023 at 9:37
  • $\begingroup$ @DemetryPascal This has actually resolved my issue! $\endgroup$ Feb 27 at 17:57

1 Answer 1

1
$\begingroup$

Underflow is happening because the result of that calculation is a number of smaller absolute value than your computer can actually represent in memory.

Underflow is not directly related to the performance of the model (e.g., learning rate or local minimum). Underflow is the interplay between computer software and hardware.

I suggest rescaling the data. For example, batch normalization often stops underflow errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.