For this question, I’ll refer to the popular YouTube video by 3Blue1Brown on deep learning applied to recognition of written numbers:


The video describes a neural network with the following layers:

1) Input (individual grayscale pixels)

2) Small-scale feature determination (e.g., the top quarter of the loop in the number 9)

3) Larger-scale features (e.g., the entire loop in the number 9)

4) The output layer, which shows the probability of an input image being each of the numbers 0-9

I’m also going to try to read through the entire wiki section here, and I’m currently on the neural networks page: https://skymind.ai/wiki/neural-network

I particularly like the explanation of coefficients... “A node combines input from the data with a set of coefficients, or weights, that either amplify or dampen that input, thereby assigning significance to inputs with regard to the task the algorithm is trying to learn; e.g. which input is most helpful is classifying data without error?“

Essentially, it’s saying that each input has some level of importance to each output, which to me begs the question... are component features/hidden layers necessary at all? In the handwriting example, couldn’t every input node be connected to every output node without the use of hidden layers? The idea is that all of the high-weight input pixels or a given output would still have high weights for that output, but the network would skip the feature/aggregation stages. Is this just a matter of training efficiency (i.e., to prevent duplication by effectively extracting the same features more than once)?

Also, do the connections between various nodes need to specifically chosen so that the number of nodes and number and selection of connections are intelligently chosen?

Is it accurate to say that a sufficiently deep neural network looks to find the significance of all relevant combinations of input values and that’s basically all it’s doing?


Every input node can be connected to every output node, but that wouldn't account for non linearity of features and relationships amongst each other that are predictive. A neural network without any hidden layers, is just a regression. I think this is a great intro to deep learning. https://www.datacamp.com/courses/deep-learning-in-python

In answer to your 2nd question: The number of nodes and connectivity are hyperparameters that should be optimized. Also, people use "drop out" (ie dropping out nodes) as a form of regularization. https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5

| improve this answer | |
  • $\begingroup$ Okay, so the network attempts to assign a value to each feature that approximates a boolean value (i.e., close to 0 or close to 1), which I assume makes it easier to get the final values to sum to 1. Without this, if the network were fed a 9, the 9 output node might register a 98% but the 4 output node might register an 85% since they are similar? Obviously, something like 98%-1% would be preferable to 98%-85%, but couldn’t the network just choose the highest percentage? $\endgroup$ – SuperCodeBrah Apr 15 '19 at 17:50
  • $\begingroup$ The network learns weights for each layer. So for the first layer (the input layer), it learns weights for each feature but these weights are for constructing the next layer. The first hidden layer is then a collection of features that are linear combinations of the input features. If there is only one hidden layer, these "new" features will each have a weight as well and you can think of that as the features that are then directly used to predict the output. The weights do not have to be 0 to 1. You just want the predictions to end up being a probability between 0 and 1, not the weights $\endgroup$ – fractalnature Apr 15 '19 at 18:26
  • $\begingroup$ I am assuming by final values you mean the predictions? Each sample gets a prediction of its own. They dont need to sum to 1. It could be a good idea to read up on logistic regression $\endgroup$ – fractalnature Apr 15 '19 at 18:28
  • $\begingroup$ I didn’t mean that the weights would be between the 0 and 1 but the result of the activation function for each neuron should be (the video says that around the 10 minute mark). So essentially, if each hidden neuron represents a feature, a deep neural net aims to determine if the feature exists (value close to 1) or not (value close to zero), but if the purpose of the network is just to assign the highest value to the correct option with no concern about minimizing the values of incorrect options, I don’t see why the intermediate layers are necessary. $\endgroup$ – SuperCodeBrah Apr 15 '19 at 20:07
  • $\begingroup$ So, actually the activation function doesn't need to be sigmoid. It is more common to use ReLU for the activation function. its x if x is positive, or 0 if it is negative (ie the bounds are [0, infinity)). en.wikipedia.org/wiki/Rectifier_(neural_networks). Actually it is not good to use the sigmoid function for each layer as that results in a vanishing gradient. towardsdatascience.com/… $\endgroup$ – fractalnature Apr 15 '19 at 20:44

There’s a brilliant free interactive book here that explains how neural networks work if you want to understand them in more detail. The chapter I have linked to demonstrates that as long as there is at least one hidden layer, neural networks can approximate any function.

As fractalnature says above, if there were no hidden layers each of your output neurons would actually be a generalised linear model that linearly combines the input features. Your neural network would effectively be a one vs rest classifier. In many cases this would perform well, and it would certainly be easier to train than a neural network, but it wouldn’t be able to achieve the same performance as it wouldn’t be able to model non linear relationships between the features.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.