Why is this ANN not working?

Question

I have created a Tf.Sequential model to when given an input of a number, get either a 1 if that number is bigger than 5 or 0 if it is not.

const model = tf.sequential();

      model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
      model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([[1], [2], [3], [4], [5], [6], [7], [8], [9]]);
      const ys = tf.tensor2d([[0], [0], [0], [0], [0], [1], [1], [1], [1]]);
      model.fit(xs, ys);
      model.predict(xs).print();

With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase, or the outputs are too high...

I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input with a bias that would point towards 5 being the "decisive" number. I thought that Tensorflow.js automatically assigns biases to weights, so I simply wrote this code:

const model = tf.sequential();

      model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
      model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
      const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
      const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
      model.fit(xs, ys);
      model.predict(xs).print();

Usually, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.

How do I make my ANN work, and why is what I did incorrect?

Answer 1

You're using the SGD optimizer, but you're only making one pass over the training data. I think the default learning rate for SGD is something like 0.01 or 0.001, and, with such a low learning rate, your model won't be able to learn the pattern with a single pass over the data.

To fix this, you need to do one or both of the following:

• raise the learning rate
• train the model for more epochs (an epoch is one pass over the training data)

To raise the learning rate, you'll need to create a custom optimizer:

. . . 
const learningRate = 0.5;
const sgdOptimizer = tf.train.sgd(learningRate);
model.compile({loss: 'meanSquaredError', optimizer: sgdOptimizer});
. . .

To train for more epochs, you just need to provide some additional arguments to the fit() function:

. . .
model.fit(xs, ys, {
    epochs: 25,
    batchSize: 8
});
. . .

Raising the learning rate will increase the magnitude of each weight update, allowing your model to learn more quickly, but if the learning rate is too high then the model may not converge. Raising the number of epochs allows for more weight updates, but will increase training time.