2
$\begingroup$ 
import numpy as np
import keras.models as km
import keras.layers as kl
import keras.optimizers as ko
import keras.losses as kloss

# This will cause no learning
np.random.seed(1692585618)

def f(x):
    a = x[0]* 3.141 + x[1]
    return a;

# Create a sample dataset 
# Input is (*, 2)
# Output is (*, )
x_train=np.array([[1,2], [3,4], [5,6]])
y_train=np.array([f(x) for x in x_train])

# These are required by the shape of x_train and y_train
in_dim = x_train.shape[1]
out_dim = 1

model = km.Sequential()
model.add(kl.Dense(units=3, activation='relu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim, activation='relu'))
model.compile(
  loss=kloss.mean_squared_error
, optimizer=ko.Adam(lr=0.1)
)

model.fit(x_train, y_train, epochs=500, batch_size=1, verbose=True)

Output:

Epoch 1/50
3/3 [==============================] - 1s 221ms/step - loss: 225.9046
Epoch 2/50
3/3 [==============================] - 0s 3ms/step - loss: 225.9046
Epoch 3/50
3/3 [==============================] - 0s 3ms/step - loss: 225.9046
Epoch 4/50
3/3 [==============================] - 0s 2ms/step - loss: 225.9046

...and so on (hundreds more)

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Try to add a learning rate of 0.01 or 0.1 and also reduce the number of hidden units to 3-4 and check and report if it works $\endgroup$
    – DuttaA
    Aug 10, 2018 at 12:40
  • 1
    $\begingroup$ Hmmmm stangely enough I found something perhaps interesting. When I make mystery_size = 1, then re-run, sometimes it learns and sometimes it doesn't. Must have to do with random initialization or something. I'll see if I can seed it such that it doesn't learn, then I'll try your suggestion. $\endgroup$
    – user5406764
    Aug 10, 2018 at 13:32
  • $\begingroup$ You should not use a ReLU activation in your final layer of a regression model because you will never be able to predict negative numbers. $\endgroup$
    – kbrose
    Aug 10, 2018 at 17:50
  • $\begingroup$ @kbrose - good point. Although, if we know the output is never negative (as the case is here), what is the impact of the final ReLU? $\endgroup$
    – n1k31t4
    Aug 10, 2018 at 21:51
  • $\begingroup$ Probably minimal. It just has the effect of clipping negative predictions to 0. Something you likely want to avoid in the real model, but AFAIK doesn’t explain the behavior in your question. $\endgroup$
    – kbrose
    Aug 10, 2018 at 22:47

1 Answer 1

Reset to default
1
$\begingroup$

In short

  • You are waaaaay undertraining. Increase the number of times you show the network your data. I am guessing training may take longer than you expect because typically networks train best with 0-mean data, which yours is not.
  • ReLU seems to cause problems with such a shallow network. Try increasing depth or using elu activation instead.
  • I confirmed having an activation in the last layer doesn't cause huge problems, but it is still a good idea of getting in the habit of knowing when you should and should not have an activation in the last layer.

Undertraining

I handled this by increasing the number of training examples by 10,000 times (you could increase the number of epochs instead but this results in better printing):

x_train=np.array([[1,2], [3,4], [5,6]] * 10000)
y_train=np.array([f(x) for x in x_train])[:, np.newaxis]

Problems with ReLU

The problems with ReLU can be handled in one of two ways, increase the number of layers when using ReLU, or use a different activation such as elu. Both trained just fine for me:

model = km.Sequential()
model.add(kl.Dense(units=3, activation='relu', input_shape=(in_dim,)))
model.add(kl.Dense(units=3, activation='relu'))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

or

model = km.Sequential()
model.add(kl.Dense(units=3, activation='elu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

Full working code

Below shows the code with elu, you can swap the block out for the ReLU version (shown above) instead and it prints very similar values.

import numpy as np
import keras.models as km
import keras.layers as kl
import keras.optimizers as ko
import keras.losses as kloss

# This will cause no learning
np.random.seed(1692585618)

def f(x):
    a = x[0]* 3.141 + x[1]
    return a;

# Create a sample dataset 
# Input is (*, 2)
# Output is (*, )
x_train=np.array([[1,2], [3,4], [5,6]] * 10000)
y_train=np.array([f(x) for x in x_train])[:, np.newaxis]

# These are required by the shape of x_train and y_train
in_dim = x_train.shape[1]
out_dim = 1

model = km.Sequential()
model.add(kl.Dense(units=3, activation='elu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

model.fit(x_train, y_train, epochs=3, verbose=True)
print('predicted: {}'.format(model.predict(x_train)[:3, 0]))
print('actual   : {}'.format(y_train[:3, 0]))

prints

Epoch 1/3
30000/30000 [==============================] - 1s 29us/step - loss: 3.0553
Epoch 2/3
30000/30000 [==============================] - 1s 20us/step - loss: 5.0199e-06
Epoch 3/3
30000/30000 [==============================] - 1s 20us/step - loss: 5.4414e-06
predicted: [ 5.1426897 13.420667  21.707573 ]
actual   : [ 5.141 13.423 21.705]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.