1
$\begingroup$

This common example is a simple way to dabble into the most basic neural network (i.e. two neurons, one input and one output).

I have been playing around with this using TensorFlow and keras, and I am wondering, it should be very simple for the NN to learn the weight (1.8) and bias (32) with little to no effort.

However, I find that (albeit, using only 13 examples) after running for 500 epochs I can get a weight of 1.8 but the bias is always off (29.2, for example). Obviously there is no activation function, given this linear problem.

My question is, for such a simple case, will a NN be able to solve this problem exactly?

$\endgroup$
  • 1
    $\begingroup$ Yes it should. Did you check the learning rate? Maybe it is not able to converge in the constant for some reason. $\endgroup$ – Carlos Mougan Mar 1 at 21:25
  • 1
    $\begingroup$ It did! Simply, 500 epochs was not enough. I ran it for 10,000 for good measure. It converged rather nicely. Thanks for the reassurance. Also, the learning rate was set to 0.01 for this example (using Adam). $\endgroup$ – Shinobii Mar 2 at 2:35
0
$\begingroup$

Hard to say without seeing code, but I can imagine several reasons:

  • You are using an activation function. You want 'linear' or no activation
  • Learning rate is much too low
  • Wrong network (should be one dense layer with one weight)
  • Poor batch size

This works pretty instantly:

from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Nadam

model = Sequential([Dense(1, activation=None, input_shape=[1])])
model.compile(loss='mean_squared_error', optimizer=Nadam(lr=0.1), metrics=['mean_absolute_error'])

F = list(range(-200,200))
C = [(f - 32) * 5 / 9 for f in F]

model.fit(F, C, batch_size=16, epochs=50)

print(model.predict(np.array([300])))
print(model.layers[0].get_weights())
[[148.88893]]
[array([[0.5555556]], dtype=float32), array([-17.777737], dtype=float32)]

300 F -> 148.8889 C, good, and it has learned the conversion is C = 5/9 * F - 17.7778, or C = 5/9 * (F - 32)

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.