Back Propagation Using MATLAB

Question

I am new to neural networks. I tried coding the backpropogation alogrithm and tried running it on a test set which gave wrong results. I have used the following knowledge to code it,

For the forward pass, $$z^l = w^la^{l-1} + b^l$$ $$a^l = g^l (z^l)$$

For the backward pass, (Here $\circ\text{ - Element wise Product}$)

For the last layer, $$\delta ^L = (a^l - y) \circ g'^L(z^L)$$ $$\frac{\partial L}{\partial w^L} = \delta^L (a^{L-1})^T$$ $$\frac{\partial L}{\partial b^L} = \delta^L$$

For the other layers, $$\delta ^l = (w^{l+1})^T(\delta^{l+1}) \circ g'^l(z^l)$$ $$\frac{\partial L}{\partial w^l} = \delta^l (a^{l-1})^T$$ $$\frac{\partial L}{\partial b^l} = \delta^l$$

For the Update, $$W : = W - \alpha \frac{\partial L}{\partial w^l} $$ $$b : = b - \alpha\frac{\partial L}{\partial b^l} $$

The following is the code which I have written,

    X = load('iris.csv');    
    W1 = rand(3,4);  
    W2 = rand(1,3);
    b1 = rand(3,1);
    b2 = rand(1,1);
    y = [ones(1,50) 2*ones(1,50) 3*ones(1,50)]';

    for j = 1:100    %for epoch
    for i = 1:150    %for iteration through dataset
    x1 = X(i,:)';
    z1 = W1*x1 + b1;
    a1 = tanh(z1);
    z2 = W2*a1 + b2;
    a2 = relu(z2);

    dz2 = (a2 - y(i)).*reluGradient(a2);  %this is delta L
    dw2 = dz2*transpose(a1);              %The derivative term of Lth
    db2 = dz2;                            %for the bias

    g1 = 1 - a1.^2;                       %Derivative 
    dz1 = transpose(W2)*dz2 .* g1;        %FOr delta l
    dw1 = dz1*transpose(x1);              %derivative term for lth layer
    db1 = dz1;                            %bias update


    W1 = W1 - 0.1*dw1;
    b1 = b1 - 0.1*db1;

    W2 = W2 - 0.1*dw2;
    b2 = b2 - 0.1*db2;

    end
    end

I am trying to train the net for the iris data set (150 X 4 - dataset Size). I have considered 4 input units, 1 hidden layer with 3 hidden units and 1 output unit. Hence the dimension of the weight matrix for first layer is 3 X 4 and for the last layer is 1 X 3. When I try to test the network I always get the input classified to class 3. I tried changing the hyper parameters, but it seems there is something wrong with the code.

In the code , I first load the CSV file, and then initialize the weight matrices accordingly. I have run two for loops one for the epoch and other for the iteration. I do a forward pass first using the above equations then a backward pass. I have made functions for the RELU and RELU_GRADIENT.

For relu:

function g = relu(z)
g = max(0,z);
end

For relu gradient:

function z = reluGradient(z)
z(z>=0) = 1;
z(z<0) = 0;
end

I would be obliged if someone can direct to me to the solution to this problem

About IRIS.csv:

Has four input features and 150 data samples. 50 of each class. Hence there are 3 classes.

Updates:

The cost function here I have used is the quadratic cost function:

$$C = \Sigma (a^L - y)^2 $$

Hence all the derivatives have been calculated with respect to that cost function. For example for the last layer,

$$\frac{\partial C}{\partial z} = (a^L - y) \circ g'^L(z^L)$$ which is $\delta^L$

Also,

I have tried using the sigmoidal transfer function instead of the relu transfer function with all the proper changes at all derivatives,weight and output vectors. (The output vector was then of (3,1) dim) As 3 classes hence - $[1,0,0]^T$ , $[0,1,0]^T$ and $[0,0,1]^T$

Thanks

Answer 1

Your problem is a classification problem with 3 classes. The appropriate function for it is softmax function, whereas you are using Relu. Another thing is that you need a cost function. The function that you optimize and the one you do calculate derivatives of with respect to parameters.

The cost function and last layer activation function must match each other. So first you choose appropriate "last" activation funtion for your problem and then cost function.

For iris you can use softmax function with cross entropy function. The derivatives of these you can easily find. :)

Another thing are the dimensions. The result of softmax function is the vector of the size (number of classes, 1) for each observation whereas you have (1,3) dimensions.

Regarding the backpropagation algorithm for the other layers it is looks ok, but the last layer equation is wrong and should be like the one below:

where C is the cost function and we calculate derivative of C with respect to a (activation of last layer) and multiply element-wise by derivative of a (here it should be softmax function with respect to z). dw and db for last layers have correct equation.

I hope this hints will help you come up with final neural network implementation for iris classification problem.