# Neural Network written in Kotlin working for simple math problems but not for MNIST classification

I'm building a neural network in Kotlin while reading the book "Neural Networks and Deep Learning" from Michael Nielsen.

At the moment the network uses: sigmoid neurons, backpropagation, stochastic gradient descent, $$1 \over \sqrt {n_{in}}$$ standard deviation for weight initialization, L2 regularization, and cross-entropy. I also created a plotter to monitor the learning process.

Here is how the network behaves with a simple problem, finding the bigger of two numbers:

Pretty good, we have a peak of 99.99%.
I can't understand why the cost is increasing instead of decreasing, but still the accuracy is good.
Things get way worse with MNIST classification; the net is basically giving random answers:

I wonder what can change the behaviour of the network this much.
I checked the MNIST data, and it seems all right.
What problem can cause this kind of learning absence? The algorithm is broken? The hyper-parameters are wrong?
All help is appreciated.

I managed to solve the problem on my own. Turns out that the problem was indeed in the hyper-parameters, as I suspected.

I was using the same values as Michael Nielsen so it seemed strange to me that they could be wrong - by a mathematical point of view, my network is exactly the same as his one! - anyway, lowering the $$\eta$$ and raising the $$\lambda$$ (in practice, lowering the weights/biases changes on a single batch) lets the network to better understand which direction it should take to minimize the cost function.
I can now get accuracies above 85%, and I'm sure that with a bit of fine-tuning I can reach at least 95%.

I also solved the cost plot problem:

I can't understand why the cost is increasing instead of decreasing [...]

The problem was that instead of doing something like

$$C= -{1 \over n} ∑_{xj} [y_j ln(a^L_j) + (1−y_j) ln(1−a^L_j)] + {λ \over 2n} ∑_w w^2$$

I was doing

$$C= -{1 \over n} ∑_{xj} [y_j ln(a^L_j) + (1−y_j) ln(1−a^L_j) + {λ \over 2n} ∑_w w^2]$$

which doesn't make sense, since the last term equals to

$$∑_{xj} {λ \over 2n} ∑_w w^2 = n {λ \over 2n} ∑_w w^2 = {λ \over 2} ∑_w w^2$$

and so we get a wrong regularization factor for the weights.