I am using scikit-learn's
MLPRegressor to learn a model with the following arguments:
hidden_layer_sizes=(16, 16), learning_rate_init=1e-2, solver='sgd', momentum=0.9, activation='tanh', learning_rate='constant'
partial_fit() to incrementally fit the model. Occasionally I get a
FloatingPointError on this line in the
updates = [self.momentum * velocity - self.learning_rate * grad for velocity, grad in zip(self.velocities, grads)] >>> FloatingPointError: underflow encountered in multiply
I am getting an underflow error when the model is calculating how much to update parameters by multiplying learning rate, momentum, velocity, and error gradients.
Does not an underflow mean that the updates are too small? Does it imply the model has reached some local minimum? Should I take this as a hint that:
- I have exhausted my data and training has ended,
- I should up my learning rate to keep updates happening
- Something else?