0
$\begingroup$

I would like to use a feedforward NN with 3 sigmoid hidden layers to demonstrate the vanishing gradient problem. I used the Pima dataset containing 8 features and it is binary classification task.

I used the callback function in keras to obtain the weights over the 10 epochs of training. As pointed out here, one of the signs of vanishing gradients is that the weights change significantly on the higher layers, whereas hardly change on the lower layers. But from what I can see the weights on the 1st hidden layer were changing, but not those on the 2nd and 3rd hidden layer, which I suppose 3 is the topmost hidden layer. Why is that happening?

model_sig_3= Sequential()  
model_sig_3.add(Dense(3, input_dim= 8, activation= 'sigmoid', kernel_initializer= 'uniform'))
model_sig_3.add(Dense(5, input_dim= 3, activation= 'sigmoid', kernel_initializer= 'uniform'))
model_sig_3.add(Dense(5, input_dim= 5, activation= 'sigmoid', kernel_initializer= 'uniform'))
model_sig_3.add(Dense(1, activation= 'sigmoid', kernel_initializer= 'uniform'))
model_sig_3.compile(loss='binary_crossentropy', optimizer='adam', metrics= ['accuracy'])

# Call back the weights on each epoch
weights_dict = {}
weight_callback = tf.keras.callbacks.LambdaCallback \
( on_epoch_end=lambda epoch, logs:  weights_dict.update({epoch:model_sig_3.get_weights()}))
# Train model
tr_results = model_sig_3.fit(X, y, validation_split=0.2, epochs=10, batch_size=32, 
                             verbose=0, callbacks=[weight_callback])

for epoch,weights in weights_dict.items():
    print('Weights for 1st Layer of epoch #',epoch+1)
    print(weights[0])
->
Weights for 1st Layer of epoch # 1
[[ 0.0099563  -0.05566332 -0.00829425]
 [-0.03811612 -0.05764516  0.01447987]
 [-0.00600547  0.03334484  0.01311353]
 [ 0.00473308 -0.0033735   0.01038584]
 [ 0.01461237 -0.0015197   0.00531363]
 [-0.01945827  0.03186332  0.05304102]
 [ 0.01516693 -0.01097144 -0.01493565]
 [ 0.00160863 -0.01567257  0.03729546]]
Weights for 1st Layer of epoch # 2
[[ 0.00935419 -0.0597956  -0.00702211]
 [-0.04256307 -0.06547074  0.02514919]
 [-0.01027493  0.02513813  0.02170457]
 [ 0.00175666 -0.00999121  0.01429512]
 [ 0.01091276 -0.00683689  0.01080244]
 [-0.02328263  0.02377684  0.05893044]
 [ 0.01487779 -0.0123193  -0.0147467 ]
 [-0.00170635 -0.0235957   0.04444925]]
Weights for 1st Layer of epoch # 3
[[ 0.00919178 -0.06091004 -0.00630577]
 [-0.04423315 -0.06791541  0.03067824]
 [-0.01190893  0.02240719  0.02489812]
 [ 0.00053162 -0.01210102  0.01588836]
 [ 0.00957205 -0.00836726  0.01233815]
 [-0.02481438  0.02114572  0.06160083]
 [ 0.01477134 -0.01270751 -0.0146466 ]
 [-0.00295153 -0.02615475  0.04840289]]
Weights for 1st Layer of epoch # 4
[[ 0.00904657 -0.06136849 -0.00560553]
 [-0.04555156 -0.06938706  0.03567327]
 [-0.01324741  0.02083453  0.02733822]
 [-0.00045427 -0.01295545  0.0171566 ]
 [ 0.00982335 -0.00917271  0.01316232]
 [-0.02606717  0.01972387  0.0635957 ]
 [ 0.01467874 -0.01290894 -0.01455865]
 [-0.00383371 -0.02746223  0.05209889]]
Weights for 1st Layer of epoch # 5
[[ 0.0088034  -0.06162698 -0.00461482]
 [-0.04689043 -0.07067651  0.04248966]
 [-0.01507491  0.01898122  0.03110605]
 [-0.00195263 -0.01430924  0.01933157]
 [ 0.01000484 -0.00993954  0.01407887]
 [-0.0277615   0.01801132  0.06655867]
 [ 0.01455059 -0.01315093 -0.01442445]
 [-0.00512013 -0.02892363  0.05762629]]


for epoch, weight in weights_dict.items():
    print('Weights for 2nd Layer of epoch #',epoch+1)
    print(weights[2])
->
Weights for 2nd Layer of epoch # 1
[[ 0.06979176  0.06662735  0.0828975   0.07505765  0.0985996 ]
 [ 0.04308664  0.01666151 -0.00234147 -0.0067628  -0.00603962]
 [ 0.1794438   0.21273921  0.18655394  0.10799292  0.20461148]]
Weights for 2nd Layer of epoch # 2
[[ 0.06979176  0.06662735  0.0828975   0.07505765  0.0985996 ]
 [ 0.04308664  0.01666151 -0.00234147 -0.0067628  -0.00603962]
 [ 0.1794438   0.21273921  0.18655394  0.10799292  0.20461148]]
Weights for 2nd Layer of epoch # 3
[[ 0.06979176  0.06662735  0.0828975   0.07505765  0.0985996 ]
 [ 0.04308664  0.01666151 -0.00234147 -0.0067628  -0.00603962]
 [ 0.1794438   0.21273921  0.18655394  0.10799292  0.20461148]]
Weights for 2nd Layer of epoch # 4
[[ 0.06979176  0.06662735  0.0828975   0.07505765  0.0985996 ]
 [ 0.04308664  0.01666151 -0.00234147 -0.0067628  -0.00603962]
 [ 0.1794438   0.21273921  0.18655394  0.10799292  0.20461148]]
Weights for 2nd Layer of epoch # 5
[[ 0.06979176  0.06662735  0.0828975   0.07505765  0.0985996 ]
 [ 0.04308664  0.01666151 -0.00234147 -0.0067628  -0.00603962]
 [ 0.1794438   0.21273921  0.18655394  0.10799292  0.20461148]]


for epoch, weight in weights_dict.items():
    print('Weights for 3rd Layer of epoch #',epoch+1)
    print(weights[4])
->
Weights for 3rd Layer of epoch # 1
[[0.18569455 0.01758892 0.09199896 0.02311023 0.1339998 ]
 [0.10732684 0.0311887  0.16159847 0.04412133 0.12491856]
 [0.15509205 0.0119047  0.07712694 0.01436216 0.1622263 ]
 [0.09615962 0.01924689 0.07611334 0.01056303 0.12956053]
 [0.16175777 0.09745486 0.17079538 0.04393203 0.20630382]]
Weights for 3rd Layer of epoch # 2
[[0.18569455 0.01758892 0.09199896 0.02311023 0.1339998 ]
 [0.10732684 0.0311887  0.16159847 0.04412133 0.12491856]
 [0.15509205 0.0119047  0.07712694 0.01436216 0.1622263 ]
 [0.09615962 0.01924689 0.07611334 0.01056303 0.12956053]
 [0.16175777 0.09745486 0.17079538 0.04393203 0.20630382]]
Weights for 3rd Layer of epoch # 3
[[0.18569455 0.01758892 0.09199896 0.02311023 0.1339998 ]
 [0.10732684 0.0311887  0.16159847 0.04412133 0.12491856]
 [0.15509205 0.0119047  0.07712694 0.01436216 0.1622263 ]
 [0.09615962 0.01924689 0.07611334 0.01056303 0.12956053]
 [0.16175777 0.09745486 0.17079538 0.04393203 0.20630382]]
Weights for 3rd Layer of epoch # 4
[[0.18569455 0.01758892 0.09199896 0.02311023 0.1339998 ]
 [0.10732684 0.0311887  0.16159847 0.04412133 0.12491856]
 [0.15509205 0.0119047  0.07712694 0.01436216 0.1622263 ]
 [0.09615962 0.01924689 0.07611334 0.01056303 0.12956053]
 [0.16175777 0.09745486 0.17079538 0.04393203 0.20630382]]
Weights for 3rd Layer of epoch # 5
[[0.18569455 0.01758892 0.09199896 0.02311023 0.1339998 ]
 [0.10732684 0.0311887  0.16159847 0.04412133 0.12491856]
 [0.15509205 0.0119047  0.07712694 0.01436216 0.1622263 ]
 [0.09615962 0.01924689 0.07611334 0.01056303 0.12956053]
 [0.16175777 0.09745486 0.17079538 0.04393203 0.20630382]]

Some clarification on the dimensionality of the weights_dict object.

len(weights_dict)                            # 10 for 10-epochs
[len(weights_dict[ep]) for ep in range(10)]  # 8 objects in each epoch-output. 4 weights and 4 biases; 
weights_id= [0,2,4,6]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.