DeepLearning: does it make sense to have more nodes in the initial layer than inputs--for tabular data

I was looking at a couple of keras tutorials and something struck me as odd. So in these two tutorials, both on housing data, the number of nodes in the first layer of the network were larger than the number of columns in the dataset itself. NOTE: I am talking about tabular data and not image data or autoencoders. I understand that autoencoders will widen during the decoding phase, and I know that segmentation models will widen during the deconvolution phase. Both of these examples seem to be different than what I am seeing in these tutorials.

The first tutorial used data on Kings County, CA housing prices to create a simple neural network. In this case, the Kings County data only has 19 variables but the network itself has an initial dense layer of size 100 nodes. Here is the code:

def basic_model_1(x_size, y_size):
t_model = Sequential()
print(t_model.summary())
t_model.compile(loss='mean_squared_error',
metrics=[metrics.mae])
return(t_model)


In the second example I was looking at Aurelion Geron's recent book on Keras and Tensorflow. In chapter 10 he uses some different California housing data to estimate a keras neural network. Again in that case he has 8 variables in the dataset, but he starts with a Dense layer of 30.

As a caveat, I did not see any columns get converted over to one-hot encodings or some sparse formulation.

This situation where the network starts wider than the data seemed odd to me, but perhaps that is just me coming from the statistics world. Can anyone explain this? Thanks.

This might seem a little bit confusing, especially for newcomers. But the problem here is basically agreeing on some terminology.

Typical Neural Networks by definition have 3 types of layers, namely Input, Hidden and Output. Because the Input layer has no specific function other that passing the input values to the next layer, many times is not mentioned. This layer does not have any weights, biases or activations and in the vast majority of frameworks (such as Keras) you don't have to write it in you code. You only need to specify the size of that input layer with the argument input_shape=(x_size,) in the first hidden layer.

What you call in your example initial layer of 100 nodes is actually that first hidden layer. Which can be on any size. And your intuition is completely right: the number of nodes in the input layer of course has to match the size of the inputs.

Hope this clarifies.

• Ah yes, thanks so much @TitoOrt, this makes a lot more sense. I come from the stats world, so I keep thinking of these things as regressions ;). I can see how the diagram makes a lot more sense. Hmm, but I was wondering when it makes sense to have more neurons than input nodes in the first hidden layer? In my head I have this notion of the hidden layer nodes will be highly correlated since they are based on a smaller number of input nodes. But I think I have to work myself out of that intuition. May 21, 2019 at 7:25

This situation where the network starts wider than the data seemed odd to me, but perhaps that is just me coming from the statistics world. Can anyone explain this?

The 1st value does not represent count of input neurons. It represent ouput from that layer. The input_shape is input to the 1st layer. Therefore, it is correct, and your doubt is genuine but if you read the explanation below, you will find we are mis-interpreting the argument.

Here we are using a multi-layer perceptron as you might be aware: we have an input layer, some hidden layers, and an output layer. When we’re making a model, it is important the first layer needs to make the input shape clear i.e. the model needs to know what input shape to expect e.g. input_shape=(x_size,)

t_model.add(Dense(100, activation="tanh", input_shape=(x_size,)))


In the first dense layer, the activation argument takes the value tanh with input_shape coming from input layer prior to this dense layer. Note: We see that the first layer has 100 as a first value for the units argument of Dense(), which is the dimensionality of the output space and which are actually 100 hidden units. This means that the model will output arrays of shape (*, 100): this is is the dimensionality of the output space and not input. Similarly, the 2nd layer has input of 100 (no need to mention as keras maps it automatically, and 50 is output from the 2nd layer.)

t_model.add(Dense(50, activation="relu"))


The intermediate layer (2nd layer) uses the relu activation function. The output of this layer will be arrays of shape (*,50), which will be input for last layer.

t_model.add(Dense(y_size))

• This is very helpful. Thanks so much. I do have a follow up question though. I get that this is a very simple model in the picture, but if all the nodes are connected to all of the inputs, then won't they have the same value? How do the weights begin to differ between nodes--what causes that differentiation? I think that this is why the weights are initialized randomly and then adjusted by backprop. But why don't they all settle back to having the same value? Is it because of interactions between hidden nodes in different layers? May 21, 2019 at 15:51