# multilayer perceptron do not converge

I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of sin(x).

The thing is my MLP only decreases the cost for the first few iterations and will get stuck at 0.5. I have tried including momentum but it doesn't help and increasing the layers or increasing the neurons do not help at all. I am not sure why this is happening ....

The summary of my code is 1) I normalize my input data either using min-max or zscore

2) Initialize random weights and bias within the range of -1 to 1

for i = 1:length(nodesateachlayer)-1
weights{i} = 2*rand(nodesateachlayer(i),nodesateachlayer(i+1))-1;
bias{i} = 2*rand(nodesateachlayer(i+1),1)-1;
end


3) then I do a forward pass where input is multiplied by weights and added with bias then activated by a transfer function (sigmoid)

for i = 2:length(nodesateachlayer)
stored{i} = nactivate(bsxfun(@plus,(weights{i-1}'*stored{i-1}),bias{i-1}),activation);
end


4) then calculate the error then do a backward pass

dedp = 1/length(normy)*error;
for i = length(stored)-1:-1:1
dpds = derivative(stored{i+1},activation);
deds = dpds'.*dedp;
dedw = stored{i}*deds;
dedb = ones(1,rowno)*deds;
dedp = (weights{i}*deds')';
weights{i}=weights{i}-rate.*dedw;
bias{i}=bsxfun(@minus,bias{i},rate.*dedb');
end


5) I have the cost plotted out at every iteration to see the descent

I assume there is something wrong with the code so where could the error possibly lies in

• Have you tried making any predictions on hold-out data? Which value range do you use? – n1k31t4 Sep 12 '18 at 10:03
• I think something has to be wrong inside your backprop implementation. The code you uploaded is quite a lot and I doubt many people here will read it. Can you outline the important parts of your implementation and explain? E.g. how you calculate your gradients and update the weights – André Sep 12 '18 at 10:07
• @n1k31t4 I did not. Because the cost always descends until around 0.5 and stop descending – userFarkill Sep 12 '18 at 13:33
• @André thanks for the suggestions. Yes i am also thinking there's something wrong that's why I re-coded the entire MLP. At first I thought is the cost which I define wrongly since it suppose to be singular value and in my code, it's an array. After modification, the same thing still happen. and I have no idea where the hell went wrong – userFarkill Sep 12 '18 at 13:35
• What's the value you set for nodesateachlayer? It seems you didn't share the code to run your functions. – user12075 Sep 12 '18 at 15:14

My view on your question, is that tiny networks seldom work. The above method uses a Neural Network to learn the function $$y=\sin(x)$$. Although this problems seems simple, it cannot be expected to be solved by a really tiny network (the above model uses a 5-layer MLP with hidden size [5,6,7], which is small).

Even if back-propagation is implemented correctly, would the model learn anything? No. I suppose Tensorflow implemented back-propagation correctly, here is the result using Tensorflow: You see, it learns almost nothing. In fact, the MSE loss is very close to 0.5 as stated above.

My suggestion is to try a 3 layer MLP with hidden size 256. Here is the result: You can see it's much better. MSE<0.1 now.

------------------code---------------------

x_ =np.atleast_2d(np.arange(0,360,1)).T
y_ = np.atleast_2d(np.sin(x_/180*np.pi))
g = tf.Graph()
with g.as_default():
with tf.variable_scope("mlp"):
input_x = tf.placeholder(shape=[None, 1], dtype=tf.float32)
input_y = tf.placeholder(shape=[None,1], dtype=tf.float32)
layer1 = tf.layers.dense(inputs=input_x, units=256, activation=tf.nn.sigmoid)
#layer2 = tf.layers.dense(inputs=input_x, units=6, activation=tf.nn.sigmoid)
#layer3 = tf.layers.dense(inputs=input_x, units=7, activation=tf.nn.sigmoid)
output_y = tf.layers.dense(inputs=layer1, units=1) # inputs=layer1
loss = tf.losses.mean_squared_error(input_y, output_y)