Why does my Multilayer Perceptron only classify linearly?

Question

I wrote a multilayer perceptron in Python. I'm trying to get it to do nonlinear classification. It has two hidden layers, so it should be perfectly capable. Unfortunately, it only seems to be able to classify linearly. I am hesitant to call this "underfitting", because I am yet to properly achieve non-linear classification, even a non-linear classification that is an underfit. In the below graph, I attempted to classify points above and below the sine function, but it just drew a straight line through the dataset.

Points of uncertainty (outputs around 0.5) are in yellow.

I am using the sigmoid as the activation function, which is simply 1/(1+((math.e) ** (-sum))).

I use two input nodes, twenty hidden nodes in two hidden layers (ten each) and one output node. I have a bias on each hidden layer and one on the output node. My input is not normalized.

I feedforward hundreds of random points and calculate the difference between the output and desired. I calculate the mean-squared error and preserve the lowest mean-square error in each generation of my evolutionary algorithm. I won't put that on display here, since it seems to work pretty well in getting the error low, although my MSE function might be incorrect.

d=0
error = 0
while d < (tPoints): # while all the points in the list have not been feedforwarded
    output1 = feedForward(n, [xValsAbove[d], yValsAbove[d]]) # feedforwards a point above the function, returns output (0,1)
    output2 = feedForward(n, [xValsBelow[d], yValsBelow[d]]) # feedforwards a point below the function, returns output (0,1)

    if output1 < 0.99:
        error += (output1)**2
    if output2 > 0.01:
        error += (1-output2)**2
    d+=1
error = error/tPoints
return error # return mean-squared error

My neurons do linear regression lines just fine. Since I have an MLP with a few layers, I wrote an algorithm for feedforwarding. It just takes the outputs of the previous layer as inputs to the next layer.

def feedForward(n, inputs): # (inputs, inputList, hidden1, hidden2, outputs, bias):

    inputList = n.getNeuronList()[0]
    hidden1 = n.getNeuronList()[1]
    hidden2 = n.getNeuronList()[2]
    outputs = n.getNeuronList()[3]
    bias = n.getNeuronList()[4]

    FFInputs = []
    FFHidden1 = []
    FFHidden2 = []
    FFOutputs = []

    for inp in inputList:
        FFInputs.append(inp.feedForward(inp.getWeights(), inp.getInputs()))
    for hidden in hidden1:
        FFHidden1.append(hidden.feedForward(hidden.getWeights(), FFInputs, bias[0]))
    for hidden in hidden2:
        FFHidden2.append(hidden.feedForward(hidden.getWeights(), FFHidden1, bias[1]))
    for out in outputs:
        FFOutputs.append(out.feedForward(out.getWeights(), FFHidden2, bias[2]))

    return FFOutputs

Graph done with matplotlib.

Does anyone experienced in ANNs have any idea why it would do this? Should I only use one layer? I am evolving the weights with an evolutionary algorithm, so should I just do more generations? Should I use a different activation function?

Answer 1

Your error calculation looks wrong:

if output1 < 0.99:
    error += (output1)**2
if output2 > 0.01:
    error += (1-output2)**2

If I am reading this correctly, output1 = 1.0 is a correct classification, since you treat any value above 0.99 as no error at all. However, you then measure the squared error as being output1 ** 2 which means the largest error is at 0.99. This is going to confuse the network, it will treat classifications above the line as perfect 1.0 if they score > 0.99, or otherwise want to adjust weights to make the classification 0.0. The opposite is true for your scoring of points under the line.

You need to either

1. Swap the conditionals (change < 0.99 to > 0.01 and vice-versa)

or

2. Replace (output1) with (1-output1) and (1-output2) with (output2)

in order to make the logic self-consistent. Which you choose depends on whether above the line points are positive class, or it is the negative ones. If above the line is positive you should use option 2.

In addition, as comments have pointed out, you might be a little ambitious with your approach. I spot the following things that make your job harder:

• Using genetic algorithm search. Whilst it is possible to train small NNs like yours using a GA, it is less efficient. GAs might be a good choice for control systems or for a-life scenarios, but are far behind cutting edge for supervised learning.

• Using mean squared error metric for classification. You should use logloss, which more heavily penalises bad guesses. With a mean squared error, there is a higher chance the network will settle for a few bad misclassifications if it also gets some other reasonable ones.

• Ambitious function to separate. A neural network should be able to classify your function, but that's a lot of curvature to learn for a starting problem. Reduce it to just a couple of cycles to start with should help you debug practical issues.

• No input normalisation. This may lead to saturated values in the network - close to 1.0 or 0.0, making it very hard to discriminate between good and bad weights. You don't show your weight initialisations - large initial weights can also cause this effect (although using a GA may actually help here, depending on how you are mutating weights).