I am implementing single layer neural network using stochastic gradient descent. When I train the model for single input it gives the answer correctly. Now when I use the second input to update my weights two things are happening:

  1. Weights obtained from previous iteration are updated due to which my previous answer is lost.

  2. Weights obtained from this cycle are not able to give correct output. If this keep happening how I can fit the whole data set?

My weight updating routine is as follows:

$W = W - (alpha) * (input) * (delta)$


I have trained my model upto 1000 iterations. Now when i predict the output using training data it only shows 80% accuracy which is same as accuracy obtained with test data. Why my trained model not converging or may be overfitting?

input_nodes = 4

hidden_nodes = 10

output_nodes = 2

learning_rate = 0.1

  • 2
    $\begingroup$ Assuming backprop is correct. Then also, it will go this way only. The solution has to satisfy both the point in the best possible way. So it should fall somewhere mid-way. You should pass more data and measure some mean metrics. $\endgroup$
    – 10xAI
    Commented Jan 25, 2021 at 16:03
  • $\begingroup$ I think the reason it is not converging is that the small network are not that expressive i.e they do not express the noise in their weights. $\endgroup$ Commented Jan 26, 2021 at 13:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.