0
$\begingroup$

The neural network I am trying to evolve uses the tanh as an activation function in each neuron and has a topology of 1-5-1, so I need at least 5 weights. The solution of the GA is a real-number vector of length 5, which represents the weights of the network and each weight should take values between -5 to 5. I wrote an R function to use as a fitness function which returns the mean squared error (MSE) of the output data in comparison to the desired output. I want it to learn the cubic function. The input data I am using is

input<-seq(-1,1,0.02)

and the output data is its cubic function

des_out<-input^3

The evaluation function is the following

evalnn<-function(x){

mse<-0
for(i in 1:length(input)){

     nn.out <- tanh(x[1]*input[i]) + tanh(x[2]*input[i]) + tanh(x[3]*input[i]) + tanh(x[4]*input[i]) + tanh(x[5]*input[i]) 
     mse <- (des.out[i] - nn.out)^2 + mse

}
return(-(mse)/length(input))
}

I have set the return value to negative, because I want the smallest value to be thought as the best fit.

gann<-ga(type="real-valued", fitness=evalnn,min=c(-5,-5,-5,-5,-5),max=c(5,5,5,5,5),popSize=100,maxiter=150,pmutation=0.01,pcrossover=0.8)

What I always get back from the GA are weights that make up a linear function although I have been experimenting with the ga's parameters quite a lot, i.e. I have tried all crossover methods and mutations. https://cran.r-project.org/web/packages/GA/GA.pdf. enter image description here
The cubic function plotted together with my output function.

I have used a linear function as training data before the cubic with this implentation and worked. It has trouble with the non-linear.

If anybody could figure out why do I get this, is there something I have missed. Thank you

$\endgroup$
  • $\begingroup$ You use the tanh function in the hidden layer, shouldn't it be logical to use sigmoid instead, because your output-values are all in the range [0-1] and tanh is in a range of [-1 1] And maybe also a sigmoid function in the output layer? Do you have still our final code and can you share this here? $\endgroup$ – user32991 Jun 3 '17 at 10:04
  • $\begingroup$ Could you share your code when you include the topology of the neural network? At this moment, I am trying to develop a forecasting model using neural network. the problem is I don't know how to optimise the architecture of my network. I hope your method could help me with it. Thank you. $\endgroup$ – ISKANDAR Dec 12 '17 at 5:24
  • $\begingroup$ Can you please provide the rationale for the requirement that the weights should lie between-5 and 5? $\endgroup$ – aivanov Dec 12 '17 at 12:53
-1
$\begingroup$

I finally got the results I was looking for.

Apparently a single hidden layer neural network cannot learn the cubic function. Instead of just evolving the weights, I included the topology of the neural network into the chromosome. My neural network can have maximum 5 hidden layers with maximum 5 neurons in each layer and the chromosome contains information for the structure of the network and the selection of the weights.

When I ran the GA with population of 200 and for 100 iterations I got a full topology of:

1(input) - 4-2-3-2-2 - 1(output), a mean squared error of 0.0051 and the following plotenter image description here

I would be glad if anyone has any comments or questions.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.