This question uses the following code:

Xtrain = np.random.rand(400,1)
ytrain = f(Xtrain)
Xval = np.random.rand(200,1)
yval = f(Xval)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    #tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(1, activation='relu')


model.fit(Xtrain, ytrain, epochs=500, verbose=0)

Using the command model.layers[0].get_weights() I get the following output:

[array([[-0.43412966, -0.51346564, -0.14263666,  0.8693182 , -0.4930619 ,
          1.249465  , -0.3924656 , -0.48984256, -0.55827504,  0.11134321]],
 array([ 0.        ,  0.        ,  0.        ,  0.34663308,  0.        ,
         0.36201355,  0.        ,  0.        ,  0.        , -0.11139664],

And using model.layers[1].get_weights() I get this:

        [ 0.19533908],
        [-0.2295354 ],
        [ 0.903574  ],
        [ 0.7272965 ],
        [ 0.02008992],
        [-0.29846227]], dtype=float32),
 array([0.29466572], dtype=float32)]

I don't understand why the outgoing weights from layer 0 (second array in the model.layers[0].get_weights() list) don't match the incoming weights for layer 1 (first array in the model.layers[1].get_weights() list). And why is there an outgoing weight from layer 1? Isn't that supposed to be the final layer?


Expected shape of parameter arrays

Each layer has two arrays:

  • one for the weights, which has a shape of (num_inputs, num_outpus)
  • one for the biases, which has a shape of (num_outputs)

Here the num_inputs is the number of input features to that layer and the num_outputs is the number of outputs from that layer (this is what you select when instantiating a layer).

Output of .get_weights()

Another important thing to note is the usage of .get_weights(). In fact there are to ways to use it:

  • From the model, i.e. model.get_weights(): This will return a flattened list containing all parameter arrays in order. For example, it could look like this: [layer1_weights, layer1_biases, layer2_weights, layer2_biases, ...]

  • From a layer, i.e. layer.get_weights(): This is what you used. Here it will return the parameter arrays for a given layer. For example model.layers[1].get_weights() will return the parameter arrays for layer1. If layer1 has biases then this will return two arrays, one for the weights and one for the biases.

I took the liberty of changing your code a bit to make this a bit more clear.

import numpy as np
import tensorflow as tf

f = lambda x: 2*x

Xtrain = np.random.rand(400, 5)  # 5 input features
ytrain = f(Xtrain)
Xval = np.random.rand(200, 5)  # 5 input features
yval = f(Xval)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),  # this layer has 5 inputs and 10 outputs
    tf.keras.layers.Dense(1, activation='relu')  # this layer has 10 inputs and 1 output


model.fit(Xtrain, ytrain, epochs=1, verbose=0)

# I will be calling .get_weights() directly from the model,
# so we expect 4 arrays: 2 for each layer.
print('First layer weights:', model.get_weights()[0].shape)
print('First layer biases:', model.get_weights()[1].shape)
print('Second layer weights:', model.get_weights()[2].shape)
print('Second layer biases:', model.get_weights()[3].shape)

The output:

First layer weights: (5, 10)  
First layer biases: (10,)  
Second layer weights: (10, 1)  
Second layer biases: (1,)
  • 1
    $\begingroup$ Thanks so much for the clear answer! $\endgroup$ – user9343456 Jul 14 '20 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.