So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data for that particular input in training data set.
x1 x2 x3 y 10 15 20 0 20 7 10 1 5 10 4 0
So imagine this is an example training set, we send these inputs to activation function, and for the first input it returns correct output (0).
But for second output it again returns 0, so the weights are adjusted until the activation function returns 1.
So now my doubt is, if the new updated weights returns the wrong output for the third input, its weight gets changed again, but will there be a situation where these weights will not satisfy for the previously tested inputs, like for example the first input in this case.
Is there a chance that new weights will return 1 for the first input, which is wrong?