in most cases it is probably the other way round but...

I have implemented a basic MLP neural network structure with backpropagation. My data is just a shifted quadratic function with 100 samples. I have created a network for regression with 1 hidden layer with 10 neurons with sigmoidal activation function and linear on the output. My algorithm is working perfectly fine, after enough epochs I get MSE around 10, which is good. On the other hand, when I try to train the same model using keras, it just cannot go below 1800 MSE - no matter what optimizers, what learning rate and how many epochs I use.

Thats my model architecture which is super simple:

model = Sequential([
    Dense(10, input_dim=1, activation='sigmoid'),
    Dense(1, activation='linear')
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(X_test, y_test, epochs=10000)

I plotted the predictions: Blue - original test set, Green - predictions from my custom network, Orange - keras predictions.

enter image description here

My question is what am I doing wrong? How is it possible that NN implementation from keras cannot properly fit such a simple function?

** When I get weights from my model and set them in Keras model it gives the exact same output on test set, so my architecture (and perceptron implemenation) is the same as one from Keras.

Edit: Solution by @brewmaster321 unfortunately does not work. Here is the plot of MSE loss for 10000 epochs: enter image description here

  • $\begingroup$ hi @tymoteusz-urban, welcome to the forum. If you find the answers to your question useful, please consider upvoting them, and accepting one by checking the tick mark next to it. If you don't find them helpful, please clarify in a comment why. $\endgroup$ Feb 28 at 9:09
  • $\begingroup$ just out of curiosity, the activation function in middle is 'sigmoid'. I thought sigmoid is more for classification? Perhaps try relu? $\endgroup$
    – lpounng
    Feb 29 at 3:54
  • $\begingroup$ How are you doing your data generation and test-train splits? $\endgroup$ Feb 29 at 7:51

1 Answer 1


In this instance, the default learning rate for the adam optimizer is 0.001 in keras, so it will take a very very long time to converge. If you bump this up as follows:

optimizer = keras.optimizers.Adam(learning_rate=0.1)
model.compile(optimizer=optimizer, loss='mean_squared_error')
model.fit(X, y, epochs=500 )

It converges after ~500 or so epochs. You can get even better results by setting the learning_rate to 0.01 and running for a couple of thousand epochs, but it will take longer. Note that you are approximating a non-linear function with a piecewise-linear approximation, and your final activation function is linear, i.e. passthrough.

import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers


# generate some quadratic function
X = np.random.rand(100,1)*20 - 10
y = X*X
plt.plot(X, y, 'o')

X_train = X[:-20]
y_train = y[:-20]
X_test  = X[-20:]
y_test  = y[-20:]

from keras import Sequential
from keras.layers import Dense

model = Sequential([
    Dense(10, input_dim=1, activation='sigmoid'),
    Dense(1, activation='linear')
# note: a larger learning rate will converge faster, a smaller learning will converge better
optimizer = keras.optimizers.Adam(learning_rate=0.1)
model.compile(optimizer=optimizer, loss='mean_squared_error')
model.fit(X_train, y_train, epochs=500 )

y_pred = model(X_test)
plt.plot(X_test, y_pred, 'o')
plt.plot(X_test, y_test, 'o')

enter image description here

  • $\begingroup$ Well, the problem is it still does not converge. I tested number of epochs from 500 to 100000, learning rate from 0.1 to 0.0001, batch_size from 1 to 100 (size of my data). I am confused as this code is super simple and should work on such simple dataset (my own implementation works perfectly). I will post a plot of history loss. $\endgroup$ Feb 28 at 20:07
  • $\begingroup$ Hm, using your exact model and changing just the learning rate, I get convergence. The only other code is generating the data and doing the test-train split. I'll edit my answer and post the entire code. $\endgroup$ Feb 29 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.