Why does my Multilayer Perceptron only classify linearly?

I wrote a multilayer perceptron in Python. I'm trying to get it to do nonlinear classification. It has two hidden layers, so it should be perfectly capable. Unfortunately, it only seems to be able to classify linearly. I am hesitant to call this "underfitting", because I am yet to properly achieve non-linear classification, even a non-linear classification that is an underfit. In the below graph, I attempted to classify points above and below the sine function, but it just drew a straight line through the dataset.

Points of uncertainty (outputs around 0.5) are in yellow.

I am using the sigmoid as the activation function, which is simply 1/(1+((math.e) ** (-sum))).

I use two input nodes, twenty hidden nodes in two hidden layers (ten each) and one output node. I have a bias on each hidden layer and one on the output node. My input is not normalized.

I feedforward hundreds of random points and calculate the difference between the output and desired. I calculate the mean-squared error and preserve the lowest mean-square error in each generation of my evolutionary algorithm. I won't put that on display here, since it seems to work pretty well in getting the error low, although my MSE function might be incorrect.

d=0
error = 0
while d < (tPoints): # while all the points in the list have not been feedforwarded
output1 = feedForward(n, [xValsAbove[d], yValsAbove[d]]) # feedforwards a point above the function, returns output (0,1)
output2 = feedForward(n, [xValsBelow[d], yValsBelow[d]]) # feedforwards a point below the function, returns output (0,1)

if output1 < 0.99:
error += (output1)**2
if output2 > 0.01:
error += (1-output2)**2
d+=1
error = error/tPoints
return error # return mean-squared error


My neurons do linear regression lines just fine. Since I have an MLP with a few layers, I wrote an algorithm for feedforwarding. It just takes the outputs of the previous layer as inputs to the next layer.

def feedForward(n, inputs): # (inputs, inputList, hidden1, hidden2, outputs, bias):

inputList = n.getNeuronList()[0]
hidden1 = n.getNeuronList()[1]
hidden2 = n.getNeuronList()[2]
outputs = n.getNeuronList()[3]
bias = n.getNeuronList()[4]

FFInputs = []
FFHidden1 = []
FFHidden2 = []
FFOutputs = []

for inp in inputList:
FFInputs.append(inp.feedForward(inp.getWeights(), inp.getInputs()))
for hidden in hidden1:
FFHidden1.append(hidden.feedForward(hidden.getWeights(), FFInputs, bias[0]))
for hidden in hidden2:
FFHidden2.append(hidden.feedForward(hidden.getWeights(), FFHidden1, bias[1]))
for out in outputs:
FFOutputs.append(out.feedForward(out.getWeights(), FFHidden2, bias[2]))

return FFOutputs


Graph done with matplotlib.

Does anyone experienced in ANNs have any idea why it would do this? Should I only use one layer? I am evolving the weights with an evolutionary algorithm, so should I just do more generations? Should I use a different activation function?

• What activation functions are you using? Are you normalizing the inputs? Have you tried more than 10 neurons? I get ok-results with 2x100 ReLU units on normalized data, Adadelta weight optimization and lots of epochs.. not very pleasing though. May 17, 2016 at 8:41
• I'm using sigmoid. I should probably try rectifier. May 17, 2016 at 14:32
• A couple of thoughts: In polynomial space, the sine function you drew is 21st order. I'm not convinced that the 3rd order ANN will display much curvature beyond what is shown (is it a straight line or is there some curve there) since it is a 3rd order poly trying to approximate a 21st order function. Could you maybe try sticking an x^2 function in there and test that before switching to a sine function with that high a wave number. If x^2 works, maybe try a sine function with the wave number set to include only 3 extrema and see how that works. May 18, 2016 at 4:42
• @AN6U5: An NN with non-linear transfer function such as sigmoid is not restricted to approximate only a certain order of polynomial, according to number of layers (I'm guessing you got "3rd order poly" from the number of layers?) There are limits for simple networks though - more based on number of neurons - and the rest of the advice is sound. May 18, 2016 at 15:59
• Where is your back propagation of the error? Where is the bias term? May 18, 2016 at 20:47

if output1 < 0.99:
error += (output1)**2
if output2 > 0.01:
error += (1-output2)**2


If I am reading this correctly, output1 = 1.0 is a correct classification, since you treat any value above 0.99 as no error at all. However, you then measure the squared error as being output1 ** 2 which means the largest error is at 0.99. This is going to confuse the network, it will treat classifications above the line as perfect 1.0 if they score > 0.99, or otherwise want to adjust weights to make the classification 0.0. The opposite is true for your scoring of points under the line.

You need to either

1. Swap the conditionals (change < 0.99 to > 0.01 and vice-versa)

or

1. Replace (output1) with (1-output1) and (1-output2) with (output2)

in order to make the logic self-consistent. Which you choose depends on whether above the line points are positive class, or it is the negative ones. If above the line is positive you should use option 2.

In addition, as comments have pointed out, you might be a little ambitious with your approach. I spot the following things that make your job harder:

• Using genetic algorithm search. Whilst it is possible to train small NNs like yours using a GA, it is less efficient. GAs might be a good choice for control systems or for a-life scenarios, but are far behind cutting edge for supervised learning.

• Using mean squared error metric for classification. You should use logloss, which more heavily penalises bad guesses. With a mean squared error, there is a higher chance the network will settle for a few bad misclassifications if it also gets some other reasonable ones.

• Ambitious function to separate. A neural network should be able to classify your function, but that's a lot of curvature to learn for a starting problem. Reduce it to just a couple of cycles to start with should help you debug practical issues.

• No input normalisation. This may lead to saturated values in the network - close to 1.0 or 0.0, making it very hard to discriminate between good and bad weights. You don't show your weight initialisations - large initial weights can also cause this effect (although using a GA may actually help here, depending on how you are mutating weights).

• My weights are initialized to a random float between -1 and 1. May 18, 2016 at 14:27
• @user255919: That might be OK with for the first hidden layer, but maybe a bit high for the second layer. The second hidden layer may also be a problem for GA search, due to dependencies between the layers meaning good values in first layer won't generally combine well with good values in second layer from a different genome. That means cross-over between genomes will not be as effective. May 18, 2016 at 15:51