I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of sin(x).

The thing is my MLP only decreases the cost for the first few iterations and will get stuck at 0.5. I have tried including momentum but it doesn't help and increasing the layers or increasing the neurons do not help at all. I am not sure why this is happening ....

I have uploaded the files for your reference here.

The summary of my code is 1) I normalize my input data either using min-max or zscore

2) Initialize random weights and bias within the range of -1 to 1

for i = 1:length(nodesateachlayer)-1    
    weights{i} = 2*rand(nodesateachlayer(i),nodesateachlayer(i+1))-1; 
    bias{i} = 2*rand(nodesateachlayer(i+1),1)-1; 

3) then I do a forward pass where input is multiplied by weights and added with bias then activated by a transfer function (sigmoid)

for i = 2:length(nodesateachlayer)
        stored{i} = nactivate(bsxfun(@plus,(weights{i-1}'*stored{i-1}),bias{i-1}),activation);    

4) then calculate the error then do a backward pass

dedp = 1/length(normy)*error;
    for i = length(stored)-1:-1:1
        dpds = derivative(stored{i+1},activation);
        deds = dpds'.*dedp;
        dedw = stored{i}*deds; 
        dedb = ones(1,rowno)*deds;
        dedp = (weights{i}*deds')';

5) I have the cost plotted out at every iteration to see the descent

I assume there is something wrong with the code so where could the error possibly lies in

  • $\begingroup$ Have you tried making any predictions on hold-out data? Which value range do you use? $\endgroup$ – n1k31t4 Sep 12 '18 at 10:03
  • $\begingroup$ I think something has to be wrong inside your backprop implementation. The code you uploaded is quite a lot and I doubt many people here will read it. Can you outline the important parts of your implementation and explain? E.g. how you calculate your gradients and update the weights $\endgroup$ – André Sep 12 '18 at 10:07
  • $\begingroup$ @n1k31t4 I did not. Because the cost always descends until around 0.5 and stop descending $\endgroup$ – userFarkill Sep 12 '18 at 13:33
  • $\begingroup$ @André thanks for the suggestions. Yes i am also thinking there's something wrong that's why I re-coded the entire MLP. At first I thought is the cost which I define wrongly since it suppose to be singular value and in my code, it's an array. After modification, the same thing still happen. and I have no idea where the hell went wrong $\endgroup$ – userFarkill Sep 12 '18 at 13:35
  • $\begingroup$ What's the value you set for nodesateachlayer? It seems you didn't share the code to run your functions. $\endgroup$ – user12075 Sep 12 '18 at 15:14

My view on your question, is that tiny networks seldom work. The above method uses a Neural Network to learn the function $y=\sin(x)$. Although this problems seems simple, it cannot be expected to be solved by a really tiny network (the above model uses a 5-layer MLP with hidden size [5,6,7], which is small).

Even if back-propagation is implemented correctly, would the model learn anything? No. I suppose Tensorflow implemented back-propagation correctly, here is the result using Tensorflow: fig1

You see, it learns almost nothing. In fact, the MSE loss is very close to 0.5 as stated above.

My suggestion is to try a 3 layer MLP with hidden size 256. Here is the result:


You can see it's much better. MSE<0.1 now.


x_ =np.atleast_2d(np.arange(0,360,1)).T
y_ = np.atleast_2d(np.sin(x_/180*np.pi))
g = tf.Graph()
with g.as_default():
    with tf.variable_scope("mlp"):
        input_x = tf.placeholder(shape=[None, 1], dtype=tf.float32)
        input_y = tf.placeholder(shape=[None,1], dtype=tf.float32)
        layer1 = tf.layers.dense(inputs=input_x, units=256, activation=tf.nn.sigmoid)
        #layer2 = tf.layers.dense(inputs=input_x, units=6, activation=tf.nn.sigmoid)
        #layer3 = tf.layers.dense(inputs=input_x, units=7, activation=tf.nn.sigmoid)
        output_y = tf.layers.dense(inputs=layer1, units=1) # inputs=layer1
        loss = tf.losses.mean_squared_error(input_y, output_y)
        train_op = tf.train.AdagradOptimizer(0.01).minimize(loss)
    with tf.Session() as sess:
        for epoch in range(300):
            _, loss_ = sess.run((train_op, loss), feed_dict={input_x:x_, input_y:y_})
        y_hat_ = sess.run(output_y, feed_dict={input_x:x_, input_y:y_})
        print(loss_, end='\t')
plt.plot(x_,y_, 'g', x_,y_hat_,'b')
plt.legend(['ground truth', 'predicted'])

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.