How to calculate change in Loss Function w.r.t Weights for a fully connected NN

Question

I read the article Optimal Brain Damage by Yann LeCun, John S. Denker and Sara A. Solla where the authors discuss estimating the saliency of each weight of a neural network (which they define by the change in the loss function upon perturbing the weight). The larger the saliency, the more effect the weight has on the learning.

Is there any code/API in Keras or Tensorflow to calculate such "weight saliency" for a simple fully connected neural network (with one or two hidden layers)?

Answer 1

@10xAI answer is good but seems to missing the final part in computing the gradients.

You can do this easily with TensorFlow's Automatic Differentiation:

import tensorflow as tf
import numpy as np

def CompileDNN():
    # Define architecture of Keras Model, Compile it
    return model

def loss_function():
    # Define loss function
    return

# Trained TensorFlow Keras Model
model = CompileDNN()
# model.fit(...)

X = # Inputs/features
Y = # Outcomes

# Compile another Keras Model, but untrained
model_temp = CompileDNN()

with tf.GradientTape() as tape:

    # Trained DNN weights
    weights = model.get_weights()

    # Add some small perturbation; here we add N(0, 1e-6) to each parameter
    weights = [ w + np.random.normal(0., 1e-6, size=(w.shape)) for w in weights ]

    # Set the new perturbed weights
    model_temp.set_weights( weights )

    # Compute outputs
    Y_hat = model_temp(X, training=False)

    # Compute loss and gradients
    loss = loss_function(Y, Y_hat)
    grads = tape.gradient(loss, model_temp.trainable_parameters)

grads will then give you the gradients of the perturbed weights, and you should probably wrap the gradient-block in a loop and repeat this process several times to get the expected gradient of loss with a $N(0, 1e-6)$ perturbation.

Answer 2

I doubt there is any ready-to-use facility but you can achieve this very easily with some custom code.

You can do -

1. Train your model(to the extent you want)
2. Get output using predict method on a dataset
3. Calculate the Loss using Loss function, y_true and last step output
4. Get the weight of every layer using Keras get_weights() API
5. Update weight as per your experiment using Keras set_weights() API
6. Calculate the new Loss using the same process as mentioned in step#3

Keras weights API -
You can get weights as a simple array and can update accordingly. May traverse in the loop on Layers Or access directly using array index.

Some code snippets(See reference links for detail)

Layer.get_weights()
Layer.set_weights(weights)

for layer in model.layers:
    print(layer .get_weights())

weights = model.layers[1].get_weights()[0]
biases = model.layers[1].get_weights()[1]

weights = [np.random.rand(*w.shape) for w in model.get_weights()]
# update
model.set_weights(weights)

References
Keras.IO
Stackoverflow
Stackoverflow
Stackoverflow