Momentum in neural networks is a variant of the stochastic gradient descent. It replaces the gradient with a momentum which is an aggregate of gradients as very well explained here.
It is also the common name given to the momentum factor, as in your case.
The momentum factor is a coefficient that is applied to an extra term in the weights update:
Note: image from visual studio magazine post
Beside others, momentum is known to speed up learning and to help not getting stuck in local minima.
As it is really nicely explained in this quora post, the momentum comes from physics:
Momentum is a physical property that enables a particular object with
mass to continue in it's trajectory even when an external opposing
force is applied, this means overshoot. For example, one speeds up a
car and then suddenly hits the brakes, the car will skid and stop
after a short distance overshooting the mark on the ground.
concept applies to neural networks, during training the update
direction tends to resist change when momentum is added to the update
scheme. When the neural net approaches a shallow local minimum it's
like applying brakes but not sufficient to instantly affect the update
direction and magnitude. Hence the neural nets trained this way will
overshoot past smaller local minima points and only stop in a deeper
Thus momentum in neural nets helps them get out of
local minima points so that a more important global minimum is found.
Too much of momentum may create issues as well as systems that are not
stable may create oscillations that grow in magnitude, in such cases
one needs to add decay terms and so on. It's just physics applied to
neural net training or numerical optimizations.
This video shows a backpropagation for different momentum values.
Other interesting posts
How does the momentum term for backpropagation algorithm work?
Hope it helps.