# Layer weights don't match in keras

This question uses the following code:

Xtrain = np.random.rand(400,1)
ytrain = f(Xtrain)
Xval = np.random.rand(200,1)
yval = f(Xval)

model = tf.keras.models.Sequential([
tf.keras.layers.Dense(10, activation='relu'),
#tf.keras.layers.Dense(10, activation='relu'),
tf.keras.layers.Dense(1, activation='relu')
])

loss=tf.keras.losses.MeanSquaredError()
)

model.fit(Xtrain, ytrain, epochs=500, verbose=0)


Using the command model.layers[0].get_weights() I get the following output:

[array([[-0.43412966, -0.51346564, -0.14263666,  0.8693182 , -0.4930619 ,
1.249465  , -0.3924656 , -0.48984256, -0.55827504,  0.11134321]],
dtype=float32),
array([ 0.        ,  0.        ,  0.        ,  0.34663308,  0.        ,
0.36201355,  0.        ,  0.        ,  0.        , -0.11139664],
dtype=float32)]


And using model.layers[1].get_weights() I get this:

[array([[-0.04339373],
[ 0.19533908],
[-0.2295354 ],
[ 0.903574  ],
[-0.17581558],
[ 0.7272965 ],
[-0.69347996],
[ 0.02008992],
[-0.30351916],
[-0.29846227]], dtype=float32),
array([0.29466572], dtype=float32)]


I don't understand why the outgoing weights from layer 0 (second array in the model.layers[0].get_weights() list) don't match the incoming weights for layer 1 (first array in the model.layers[1].get_weights() list). And why is there an outgoing weight from layer 1? Isn't that supposed to be the final layer?

### Expected shape of parameter arrays

Each layer has two arrays:

• one for the weights, which has a shape of (num_inputs, num_outpus)
• one for the biases, which has a shape of (num_outputs)

Here the num_inputs is the number of input features to that layer and the num_outputs is the number of outputs from that layer (this is what you select when instantiating a layer).

### Output of .get_weights()

Another important thing to note is the usage of .get_weights(). In fact there are to ways to use it:

• From the model, i.e. model.get_weights(): This will return a flattened list containing all parameter arrays in order. For example, it could look like this: [layer1_weights, layer1_biases, layer2_weights, layer2_biases, ...]

• From a layer, i.e. layer.get_weights(): This is what you used. Here it will return the parameter arrays for a given layer. For example model.layers[1].get_weights() will return the parameter arrays for layer1. If layer1 has biases then this will return two arrays, one for the weights and one for the biases.

I took the liberty of changing your code a bit to make this a bit more clear.

import numpy as np
import tensorflow as tf

f = lambda x: 2*x

Xtrain = np.random.rand(400, 5)  # 5 input features
ytrain = f(Xtrain)
Xval = np.random.rand(200, 5)  # 5 input features
yval = f(Xval)

model = tf.keras.models.Sequential([
tf.keras.layers.Dense(10, activation='relu'),  # this layer has 5 inputs and 10 outputs
tf.keras.layers.Dense(1, activation='relu')  # this layer has 10 inputs and 1 output
])

loss=tf.keras.losses.MeanSquaredError()
)

model.fit(Xtrain, ytrain, epochs=1, verbose=0)

# I will be calling .get_weights() directly from the model,
# so we expect 4 arrays: 2 for each layer.

print('First layer weights:', model.get_weights()[0].shape)
print('First layer biases:', model.get_weights()[1].shape)
print('Second layer weights:', model.get_weights()[2].shape)
print('Second layer biases:', model.get_weights()[3].shape)


The output:

First layer weights: (5, 10)
First layer biases: (10,)
Second layer weights: (10, 1)
Second layer biases: (1,)

• Thanks so much for the clear answer! – user9343456 Jul 14 '20 at 15:38