Weights in neural network

Question

So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data for that particular input in training data set.

 x1          x2           x3          y
 10          15           20          0
 20           7           10          1
 5           10           4           0

So imagine this is an example training set, we send these inputs to activation function, and for the first input it returns correct output (0).

But for second output it again returns 0, so the weights are adjusted until the activation function returns 1.

So now my doubt is, if the new updated weights returns the wrong output for the third input, its weight gets changed again, but will there be a situation where these weights will not satisfy for the previously tested inputs, like for example the first input in this case.

Is there a chance that new weights will return 1 for the first input, which is wrong?

Answer 1

In parametric models such as linear regression, logistic regression and multi-layers perceptrons, weights are updated with regards to the "difference" between the output of your model and the real label.

More precisely, weights are updated using the gradient descent / backpropagation procedure. It is composed of two parts : the forward pass and the backward pass.

For a given observation (or a set of observations), the forward pass is about feeding the model with observations and output a result a. This output "a" is then compared with the real value, the label y. Using some cost function metrics (such as Absolute Error or Square Error for regression purposes, cross-entropy for classification purposes...), we can then compute j(y,a) which is the error between the output and the real value.

We can now run the backward pass which is about computing the derivative of the cost function with regards to any weight / bias coefficient in the logistic regression / neural network. We can the update coefficients such as :

Where alpha is the learning rate. So to answer your question, weights are not updated until they reach the expected value. We are just trying to reach the minimum in cost function surface by running gradient descent procedure.

Answer 2

Given the sampled three rows, x1 and x2 look as if they can determine y, and x3 can be left unconsidered, or in other words, x1 and x2 seem to correlate with y .
As we are not discussing any semantics, do not know what these figure mean, new training data can void the model, even make y practically unpredictable, finally.

It is just an assumption, not guaranteed, that there is a significant correlation at allx1,x2,and x3 are enough data points to ćreate a model that can be used to predict y with sufficient precision.