In a neural network, is it possible to gradient descent with more than one input?

Question

I went through a few tutorials, examples recently, and all (not sure if just for demonstration purposes) done gradient descent for one input. To get a deep understanding of backpropagation, I wrote a program to do backpropagation just do understand it more deeply. In linear/logistic regression, it makes sense to do gradient descent on the average of the costs through multiple inputs, and outputs, because the there's just one layer of weights, and the inputs directly affect the outputs.

In case of neural networks, we get back a layer of activations (outputs), and we have the expected outputs with matching shape, so we're getting the costs, by subtracting the expected output, with our actual output, and we're propagating this back with the chain rule. But this way we have to compare our costs with the activations of the neurons, which are unique, and dependent on the inputs. So even if we would take a bunch of inputs, get their costs, and average the costs, how could we decide which neuron activation layer should it be compared to?

Answer 1

As to what I understand from your question, yes you can have multiple features (variables/input) and use gradient descent to minimize the loss function which has the input variables. Here is an article that gives you a basic vector calculus idea as to what exactly does a gradient is and what it does in the algorithm! https://towardsdatascience.com/wondering-why-do-you-subtract-gradient-in-a-gradient-descent-algorithm-9b5aabdf8150

Answer 2

We calculate the Loss only once per batch(average for each data point) and only at the output layer not at every Neuron.

$\hspace{5cm}$ Loss = Loss_fn(y_true, y_pred)

In other words, this Loss_fn is a function of all the weight/bias and we want to know the Gradient w.r.t each weight/bias.
We can't get this directly for each parameter as each layer is dependent on the previous layer.

Output is something similar to - f( g( h( k(x) ) ) )
This is where you apply Chain rule to get the Gradient till the first layer in a backward fashion. $\hspace{2cm}$
$\hspace{3cm}$ [Image Credit - https://leonardoaraujosantos.gitbook.io/]

For the last layer,
We can get the Gradient directly as for this layer we have got both the output error and input(output of the last layer from forward pass)

Answer 3

After weeks trying to get into the very depth of the question as someone who's not from a maths background, neither data science, I figured out how it works internally.

This is one, raw way to make it work, and my implementation, haven't seen it anywhere implemented like this, I'm pretty sure there are more modern ways to do it, so if you want, use it with caution.

Here's the code:

const SNN = ({
  trainFeatures = [],
  trainLabels = [],
  layers = [
    {
      type: 'mlp',
      size: 10,
    },
    {
      type: 'sigmoid'
    },
    {
      type: 'mlp',
      size: 12,
    },
    {
      type: 'sigmoid'
    },
  ],
  learningRate = 0.1,
  batchSize,
  standardize = false,
}) => {
  const { mean, variance } = tf.tidy(() => {
    const { mean, variance } = tf.moments(trainFeatures, 0);
    const filler = variance
      .cast("bool")
      .logicalNot()
      .cast("float32");

    return { mean, variance: variance.add(filler) };
  });

  learningRate = tf.tensor(learningRate);

  trainFeatures = standardize
    ? standardizeFeatures(trainFeatures, mean, variance)
    : trainFeatures;

  //make sure there's a last layer matching the label size, with sigmoid activation
  layers.push(
    { type: 'mlp', size: trainLabels.shape[1] },
    { type: 'sigmoid'}
  );

  layers.forEach((layer, i) => {
    if (layer.type === 'mlp') {
      const { size } = layer;
      const prevSize = (
        layers
          .slice(0, i)
          .filter(({ type }) => type === 'mlp')
          .reverse()[0]
        || {}
      ).size || trainFeatures.shape[1];

      const bias = 1;
      layer.weights = tf.truncatedNormal([
        bias + prevSize, size
      ]);
    }
  })

  const getActivationsOfLayers = featureSet =>
    tf.tidy(() => {
      const activations = [featureSet];

      layers.forEach(layer => {
        const prevLayerActivations = activations[activations.length - 1]

        if (layer.type === 'mlp') {
          const { weights } = layer;

          const currentLayerActivations = prevLayerActivations
            .pad([[0,0],[1,0]], 1) //add 1 padding for bias
            .matMul(weights);

          activations.push(currentLayerActivations);
        } else if (layer.type === 'relu') {
          const { slope } = layer;

          const currentLayerActivations = prevLayerActivations
            .leakyRelu(slope);
  
          activations.push(currentLayerActivations);
        } else if (layer.type === 'sigmoid') {
          const currentLayerActivations = prevLayerActivations
            .sigmoid();
  
          activations.push(currentLayerActivations);
        }
      });

      return activations;
    });

  const getCost = (predictionLabelSet, labelSet) =>
    predictionLabelSet
      .sub(labelSet)
      .pow(2)
      .sum()
      .div(labelSet.shape[0]);

  const gradientDescent = (activationsOfLayers, labelSet) => {
    let dCda = activationsOfLayers[activationsOfLayers.length - 1]
      .sub(labelSet)
      .mul(2);

    for (let layerIndex = layers.length - 1; layerIndex >= 0; layerIndex--) {
      const nextActivations = activationsOfLayers[layerIndex + 1];
      const activations = activationsOfLayers[layerIndex];

      const layer = layers[layerIndex];
      
      if (layer.type === 'sigmoid') {
        //rows will be activations, columns will be observations
        const dadn = nextActivations
          .sub(1)
          .mul(nextActivations)
          .mul(-1);

        dCda = dCda.mul(dadn);
      } else if (layer.type === 'relu') {
        const { slope } = layer;

        const dadn = nextActivations
          .step(slope);

        dCda = dCda.mul(dadn);
      } else if (layer.type === 'mlp') {
        const { weights } = layer;

        //rows will be activations, columns will be observations
        const dadw = activations
          .pad([[0,0],[1,0]], 1)
          .transpose();

        const dCdw = dadw.matMul(dCda);

        const newWeights = weights.sub(dCdw.mul(learningRate));

        layer.weights = newWeights;

        dCda = dCda
          .matMul(
            weights
              .slice([1, 0], [-1, -1])
              .transpose()
          );
      }
    }
  };

  const train = (iterations = 100) => {
    const batchQuantity = Math.floor(trainFeatures.shape[0] / batchSize);

    for (let i = 0; i < iterations; i++) {
      for (let j = 0; j < batchQuantity; j++) {
        const startRowIndex = j * batchSize;
        const endRowIndex = batchSize;

        const featureSet = trainFeatures.slice(
          [startRowIndex, 0],
          [endRowIndex, -1]
        );
        const labelSet = trainLabels.slice(
          [startRowIndex, 0],
          [endRowIndex, -1]
        );
        const activationsOfLayers = getActivationsOfLayers(featureSet);

        const predictionLabels =
          activationsOfLayers[activationsOfLayers.length - 1];

        const cost = getCost(predictionLabels, labelSet);
        cost.print();

        gradientDescent(activationsOfLayers, labelSet);

        tf.dispose([featureSet, labelSet, activationsOfLayers]);
      }
    }
  };

  const predict = featureSet => {
    featureSet = standardize
      ? standardizeFeatures(featureSet, mean, variance)
      : featureSet;
    const activationsOfLayers = getActivationsOfLayers(featureSet);
    const predictionLabels = activationsOfLayers[
      activationsOfLayers.length - 1
    ].softmax();

    return predictionLabels;
  };

  const test = ({ testFeatures, testLabels }) =>
    tf.tidy(() => {
      const numberOfObservations = tf.tensor(testLabels.shape[0]);

      testLabels = testLabels.argMax(1);
testLabels.print();
      const predictionLabels = predict(testFeatures).argMax(1);
predictionLabels.print();
      const incorrect = predictionLabels.notEqual(testLabels).sum();

      return numberOfObservations.sub(incorrect).div(numberOfObservations);
    });

  return {
    train,
    test
  };
};

Basically because of matrix multiplication it doesn't matter how many items are in your batch.

We update our weights with dCdw (effect of weight on Cost), and we get dCdw by matrix multiplying dadw (effect of weight on its activation) with dCda (effect of activation on Cost). In case of the last layer, dCda is (a - y) * 2 (if cost function is mean squared error), and dadw is the previous layer's activations. One of their axes will have the same shape guaranteed, and it will be the batch size. So you can have any inputs (in the question by inputs I meant batch size), because the other size of each matrices are the sizes of the weights of their layers (and I find this beautiful).

If on the one before last layer you have 3 neurons, on the last layer, you have 2 neurons, you will have a weight between them with shape [2,3] (or [3,2] depending on which direction you go). Batch size is x (to demonstrate it can be anything). One before last activations will have a shape of [x,3]. You matmul it with the weights with shape [3,2]. You will receive your last activations (prediction) with shape [x,2].

Your dCda will have a shape of [x,2]. Your dadw is the one before last activations (which had a shape of [x,3]), you transpose it, so it will have a shape of [3,x], and you matmul it with dCda ([x,2]), so you receive a matrix (dCdw) with shape [3,2] which is the shape of the weights. And we update the weights with it by multiplying with learning rate etc.

As you can see from the code, on which depth you are in your layer on neurons doesn't really matter, you just update dCda by matmulling the last dCda with your transposed weights.

I hope it will help someone, and save some time, but what I suggest to understand backpropagation is to take a simple network (like a 2-3-2), and write down one by one which activation does what. I have my notes on it, and if it would help anyone, I'm happy to share, but I don't think it's worth cutting down the path.