To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training (with gradient descent) and got excellent results.

However, I just realized that my model had far more degrees of freedom than data points. Say the data are all nn pixel pictures to recognize, so the number of degrees of freedom is nn+1. I use n=50, but I only use less than 20 data points (20 training pictures). From the point of view of linear fitting/regression, the number of data points should at least surpass that of degrees of freedom. But now in my model the reverse happens, no solution should be found. What happened

  • $\begingroup$ In neural networks it is common to have more variables than data. Anyways after training most of the weights will be close to zero. $\endgroup$
    – keiv.fly
    Dec 9, 2018 at 2:25
  • $\begingroup$ But how is the solution trustable, if in reality the solution is far from being accurate? $\endgroup$
    – feynman
    Dec 9, 2018 at 9:50
  • $\begingroup$ In neural networks there are many points which have the same minimum of the loss function. There is even a good practice when you train the network to train it 10 times to the optimum, take the best 5 and average the forecast of the 5 networks. This leads to a significant improvement in forecasting accuracy. $\endgroup$
    – keiv.fly
    Dec 9, 2018 at 10:06
  • $\begingroup$ OK. But how to explain mathematically that fitting is possible when there are more variables than data? Take some low dimension examples. To use only 1 point to fit a line (at least 2 points needed), one will get a random (up to the initial weight) line passing that 1 point. To use only 2 points to fit a plane (at least 3 points needed), one will get a random (up to the initial weights) plane passing that line connecting those 2 points. $\endgroup$
    – feynman
    Dec 9, 2018 at 10:14
  • $\begingroup$ For the optimum only the result near the points is important. What lies outside is not relevant if you get a good forecast. More dimensions than points allows you to have better non-linearity. You can zero several dimensions by having one linear combination of other dimensions and one Relu activation on the result of the combination. $\endgroup$
    – keiv.fly
    Dec 9, 2018 at 10:36

1 Answer 1


Good question. The solution lies in understanding how weights play into the model's prediction. I've glossed over a bunch of issues here, but the big picture is what is most important.


  1. When you create a model all the weights are psuedo-randomly initialized. This solves your problem of not having enough degrees of freedom to solve a linear regression. We've interpolated our weights and our model will update only the weights it needs as it trains.
  2. Additionally, your model will probably find that most of your weights converge to zero. If you are using sklearn's load_digits() function, I recall that about half of each image was just white pixels. It is likely that there is a significant percentage of pixels that do not matter across all images (and thus might be assigned a weight of zero). Your question of how can you trust the prediction of the model is thus answered and we can modify our statement about degrees of freedom to: There cannot exist a greater number of discovered statistical trends than there are data samples. It is hard to measure "number of discovered statistical trends", but it follows that there is a finite number and that they are less than the number of samples minus 1.


With a fully-connected, shallow neural network consisting of the input layer with weights and an output layer with two neurons, the input to the activation function of each of the output neurons will be something like the following:



  1. $P$ is the class probability,
  2. $B_1$ is the bias of the output layer,
  3. $W_i$ is the weight of a specific neuronal connection,
  4. ${I_i}_A$ is the output of the connected neuron if activated (in this case the pixel value), and
  5. $f()$ is the activation function of the output layer neuron.

If we didn't initialize $W_i$ to something/anything, we would be unable to solve the equation (more variables than samples). However, by randomly initializing them, the equation immediately becomes solvable and we can use stochastic gradient descent or a different algorithm to optimize our weights. I hope this shows how pseudo-randomly initializing $W_i$ solves the mathematical problem of degrees of freedom.

  • $\begingroup$ almost the same as i thought. i usually put initial weights to be 0 by default. if i change the initial weights into other values, the problem is still always solvable! but what about the resultant calculated weights? $\endgroup$
    – feynman
    Jan 30, 2019 at 15:22
  • $\begingroup$ I don't understand the problem with the equation always being solvable as we will minimize our cost function and find locally optimal weights. perhaps this video will explain neural network error minimization better than I can in a two sentence comment :P youtube.com/watch?v=Dws9Zveu9ug $\endgroup$
    – Joe B
    Jan 30, 2019 at 19:17
  • $\begingroup$ i dont think this video addresses my question $\endgroup$
    – feynman
    Feb 14, 2019 at 10:24
  • $\begingroup$ Maybe I misunderstand the question. Can you rephrase it? $\endgroup$
    – Joe B
    Feb 15, 2019 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.