Neural Network - exercise

Question

I am currently learning the concept of neural networks by myself. I am working with a very good pdf from http://neuralnetworksanddeeplearning.com/chap1.html

I also did a few exercises but there is one exercise I really don't understand.

Task: There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01. I found also the solution, as can be seen in the second image

I understand why the matrix has to have this shape, but I really struggle to understand the step, where the user calculates

0.99 + 3*0.01
4*0.01

PS: I know that the question was answered in Extra output layer in a neural network (Decimal to binary). However, the specific step I am looking for was not answered.

Answer 1

But what happens if the input has 0.99 and 0.01?

indicates the point: if the third layer is perfect, then we're already done; but we need to account for small variations in the third layer. We need to understand how far off we can get after multiplying by $W$.

Now, it appears that the answer gets this a little wrong. If $\vec{x}$ consists of one entry at least 0.99 and nine entries of at most 0.01, then $W\vec{x}$ is off by some number depending on the ten columns of $W$, not the four rows. And we need lower bounds for the correct slots, and upper bounds for the incorrect slots (rather than upper bounds for both).

For example, if $\vec{x}=\langle 0.05, 0.997, 0.1, 0.01, \dotsc\rangle$, we'd get

$$W\vec{x} = \begin{pmatrix} 0.997 +0.01 + \dotsb \\ 0.1 +0.01 + \dotsb \\ \vdots \end{pmatrix}.$$

We need to use bias/activation/cutoffs to "push" this to the desired $\langle 1,0,0,0\rangle$ (and similarly for other $\vec{x}$). Well, the entries where we desire a 1 are at least 0.99 in the relevant position and at least 0 elsewhere (seemingly the activations are supposed to make everything positive?), so we get at least 0.99. The entries where we desire a 0 are at most 0.01 in every non-zero position, so we get at most $5*0.01$ (the first row of $W$ has the most nonzero entries, 5). So setting any cutoff (with relevant a bias and/or activation function) between 0.05 and 0.99 will suffice.

[Another example, since that might have been too wordy. Say the output in the third row is $$\vec{x}=\langle 0, 0.01, 0.99, 0.01, 0, 0.01, 0.01, 0.01, 0, 0.01\rangle.$$ Then $$W\vec{x} = \langle 0.05, 1.02, 0.03, 0.01\rangle.$$ You can see how I've cooked up $\vec{x}$ to get the largest possible first entry in $W\vec{x}$ given the constraints on $\vec{x}$.]