I wrote a multilayer perceptron in Python. I'm trying to get it to do nonlinear classification. It has two hidden layers, so it should be perfectly capable. Unfortunately, it only seems to be able to classify linearly. I am hesitant to call this "underfitting", because I am yet to properly achieve non-linear classification, even a non-linear classification that is an underfit. In the below graph, I attempted to classify points above and below the sine function, but it just drew a straight line through the dataset.
Points of uncertainty (outputs around 0.5) are in yellow.
I am using the sigmoid as the activation function, which is simply
1/(1+((math.e) ** (-sum))).
I use two input nodes, twenty hidden nodes in two hidden layers (ten each) and one output node. I have a bias on each hidden layer and one on the output node. My input is not normalized.
I feedforward hundreds of random points and calculate the difference between the output and desired. I calculate the mean-squared error and preserve the lowest mean-square error in each generation of my evolutionary algorithm. I won't put that on display here, since it seems to work pretty well in getting the error low, although my MSE function might be incorrect.
d=0 error = 0 while d < (tPoints): # while all the points in the list have not been feedforwarded output1 = feedForward(n, [xValsAbove[d], yValsAbove[d]]) # feedforwards a point above the function, returns output (0,1) output2 = feedForward(n, [xValsBelow[d], yValsBelow[d]]) # feedforwards a point below the function, returns output (0,1) if output1 < 0.99: error += (output1)**2 if output2 > 0.01: error += (1-output2)**2 d+=1 error = error/tPoints return error # return mean-squared error
My neurons do linear regression lines just fine. Since I have an MLP with a few layers, I wrote an algorithm for feedforwarding. It just takes the outputs of the previous layer as inputs to the next layer.
def feedForward(n, inputs): # (inputs, inputList, hidden1, hidden2, outputs, bias): inputList = n.getNeuronList() hidden1 = n.getNeuronList() hidden2 = n.getNeuronList() outputs = n.getNeuronList() bias = n.getNeuronList() FFInputs =  FFHidden1 =  FFHidden2 =  FFOutputs =  for inp in inputList: FFInputs.append(inp.feedForward(inp.getWeights(), inp.getInputs())) for hidden in hidden1: FFHidden1.append(hidden.feedForward(hidden.getWeights(), FFInputs, bias)) for hidden in hidden2: FFHidden2.append(hidden.feedForward(hidden.getWeights(), FFHidden1, bias)) for out in outputs: FFOutputs.append(out.feedForward(out.getWeights(), FFHidden2, bias)) return FFOutputs
Graph done with matplotlib.
Does anyone experienced in ANNs have any idea why it would do this? Should I only use one layer? I am evolving the weights with an evolutionary algorithm, so should I just do more generations? Should I use a different activation function?