I am currently learning the concept of neural networks by myself. I am working with a very good pdf from http://neuralnetworksanddeeplearning.com/chap1.html

I also did a few exercises but there is one exercise I really don't understand.

Task: There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01. enter image description here I found also the solution, as can be seen in the second image enter image description here

I understand why the matrix has to have this shape, but I really struggle to understand the step, where the user calculates

0.99 + 3*0.01

PS: I know that the question was answered in Extra output layer in a neural network (Decimal to binary). However, the specific step I am looking for was not answered.


1 Answer 1


But what happens if the input has 0.99 and 0.01?

indicates the point: if the third layer is perfect, then we're already done; but we need to account for small variations in the third layer. We need to understand how far off we can get after multiplying by $W$.

Now, it appears that the answer gets this a little wrong. If $\vec{x}$ consists of one entry at least 0.99 and nine entries of at most 0.01, then $W\vec{x}$ is off by some number depending on the ten columns of $W$, not the four rows. And we need lower bounds for the correct slots, and upper bounds for the incorrect slots (rather than upper bounds for both).

For example, if $\vec{x}=\langle 0.05, 0.997, 0.1, 0.01, \dotsc\rangle$, we'd get

$$W\vec{x} = \begin{pmatrix} 0.997 +0.01 + \dotsb \\ 0.1 +0.01 + \dotsb \\ \vdots \end{pmatrix}.$$

We need to use bias/activation/cutoffs to "push" this to the desired $\langle 1,0,0,0\rangle$ (and similarly for other $\vec{x}$). Well, the entries where we desire a 1 are at least 0.99 in the relevant position and at least 0 elsewhere (seemingly the activations are supposed to make everything positive?), so we get at least 0.99. The entries where we desire a 0 are at most 0.01 in every non-zero position, so we get at most $5*0.01$ (the first row of $W$ has the most nonzero entries, 5). So setting any cutoff (with relevant a bias and/or activation function) between 0.05 and 0.99 will suffice.

[Another example, since that might have been too wordy. Say the output in the third row is $$\vec{x}=\langle 0, 0.01, 0.99, 0.01, 0, 0.01, 0.01, 0.01, 0, 0.01\rangle.$$ Then $$W\vec{x} = \langle 0.05, 1.02, 0.03, 0.01\rangle.$$ You can see how I've cooked up $\vec{x}$ to get the largest possible first entry in $W\vec{x}$ given the constraints on $\vec{x}$.]

  • $\begingroup$ Two questions: 1) "Say the output in the third row is " Do you mean old output layer? 2) Do I understand it correctly: Wx= (0.05,1.02,0.03,0.01) Adding bias would be: Assuming b=-(0.5,0.5,0.5,0.5) results in Wx+b=(0.05-0.5,1.02-0.5,0.03-0.5,0.01-0.5)=(0,1,0,0) So b is not a scalar, rather a vector?! $\endgroup$
    – SMS
    Commented Jul 1, 2019 at 9:10
  • $\begingroup$ 1) Yes. 2) Yes, bias can be specified per neuron, so treating the whole layer as a vector means b should be treated as a vector. (But in this example, we can apply the same bias everywhere, so the vector has (can have) all the same entries.) [And of course, Wx+b is actually (-0.45, 0.52, -0.47, -0.49); but we're assuming something about activation to force these to 0 and 1. The important thing is just to show that there's separation of the two classes even with the noise.] $\endgroup$
    – Ben Reiniger
    Commented Jul 1, 2019 at 14:52
  • $\begingroup$ Yes indeed that's how I "looked" on the problem, by considerung each layer as a vector. Then it makes sense to have b as a vector. Just waned to go for sure :) Thanks a lot for your help $\endgroup$
    – SMS
    Commented Jul 2, 2019 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.