0
$\begingroup$

Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the error using the gamma variables in the code. The problem is I get very small change in the average error after the first iteration. I'm using initial weights randomly between -0.1 and 0.1 and learning rate of 0.03. Inputs are roughly between -0.001 and 0.001. Target output is between -0.5 and 0.5. I just want to check if the code is correct. I've tried higher values for learning rate and initial weights and the error actually goes up and settles around 1.0. Now is the code wrong or do I just need to fine tune the initial weights and learning rate? Many thanks for your replies!

ERROR = 0.5 * MathPow( (TARGET - output), 2 );

double gamma = ERROR * ( 1 - ( output * output ) );

// HIDDEN-OUTPUT LAYER UPDATE...
for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
{
       double gamma_hi = gamma  * HIDDEN_OUTPUT_WEIGHTS[ i ];

       HIDDEN_GAMMAS[ i ] = gamma_hi;

       double hi_weight_delta = LEARNING_RATE * gamma_hi  * HIDDEN_OUTPUTS[ i ];

       HIDDEN_OUTPUT_WEIGHTS[ i ] += hi_weight_delta;
}

// INPUT-HIDDEN LAYER UPDATE

for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
{
   for ( int j=0; j<INPUT_LAYER_SIZE; j++ )
   {   
        double gamma_ij = HIDDEN_GAMMAS[ i ] * INPUT_HIDDEN_WEIGHTS[ i ][ j ];

        double ij_weight_delta = LEARNING_RATE * gamma_ij * INPUT_LAYER_INPUTS[ j ];

        INPUT_HIDDEN_WEIGHTS[ i ][ j ] += ij_weight_delta;
    }
}

// OUTPUT CODE:

// Calculate value of each hidden...
   for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
   {
      double hi = 0.0;

      for ( int j=0; j<INPUT_LAYER_SIZE; j++ )
      {
         hi += INPUT_LAYER_INPUTS[ i ]* HIDDEN_OUTPUT_WEIGHTS[ i ][ j ];
      }

      HIDDEN_OUTPUTS[ i ] = TANH( hi );
   }

   double output = 0.0;

   // Calculate value of the ouput
   for ( int i=0; i<HIDDEN_LAYER_SIZE; i++ )
   {

      output += HIDDEN_OUTPUTS[ i ] * HIDDEN_OUTPUT_WEIGHTS[ i ];
   }

   output = TANH( output );
$\endgroup$
1
$\begingroup$

It looks like you're adding the delta to the weights instead of subtracting it. Gradient descent is given by the following calculation run for some iterations:

$$ x^{current} = x^{previous} - \alpha \frac{dy}{dx} $$

where $\alpha$ is the learning rate. We subtract the value because derivatives go in the direction of steepest ascent. So the following lines:

HIDDEN_OUTPUT_WEIGHTS[ i ] += hi_weight_delta;
INPUT_HIDDEN_WEIGHTS[ i ][ j ] += ij_weight_delta;

should be

HIDDEN_OUTPUT_WEIGHTS[ i ] -= hi_weight_delta;
INPUT_HIDDEN_WEIGHTS[ i ][ j ] -= ij_weight_delta;

Additionally, I'm not sure what the architecture of your network is, but it doesn't look like you have bias units- consider adding them so that your neurons can have non-zero centered activations. Not having them may impact your network's ability to learn.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, thankyou for your reply. I have the same number of inputs as hidden nodes and just 1 hidden layer and 1 output node. I added the output code above. I find now the error goes down slightly in the order of 0.0001 in the first handful of iterations but then stops going down much at all and with higher learning rates and initial weights actually goes up with very small output values nowhere near the target. Will bias units help this much? $\endgroup$ – indigo May 28 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.