Multi-output regression problem with Keras

Question

number of features: 12 , -15 < each feature < 15

number of targets: 6 , 0 < each target < 360

number of examples: 262144

my normalization: I normalized the features so that they are between 0 and 1. I normalized the targets so that they are between 1 and 10.

This is the model that I am using:

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(6, activation='linear')
])

model.compile(optimizer="rmsprop", loss='mean_squared_error', metrics=['accuracy'])

model.fit(training_x, training_y, epochs=10, batch_size=100)

This is the best result that I have got (training):

235929/235929 [==============================] - 8s 33us/step - loss: 8.9393e-04 - acc: 0.6436

testing:

loss: 0.00427692719418488

acc: 0.033187106618348276

I get almost 0% accuracy on the test set! I need a model to solve this ML problem.

Answer 1

Accuracy is a metric for classification, not regression.

$$Accuracy = \frac{\text{Correct classification}}{\text{Number of classifications}}$$

So when you use accuracy for regression only the values where actual_label == predicted_label are evaluated as true are counted as correct classifications. That will happen quite rarely when you are doing regression, resulting in an accuracy that is close to zero.

Instead you should use something like mean absolute error or mean squared error as validation metrics for regression.

Answer 2

You don't have to normalize regression targets but in a different case you might have wanted to scale them so that the loss of one output doesn't dominate over the loss for other outputs.

The very important thing here is that you are predicting angles which are periodically bounded but you have them as continuous values. So 359 should be very close to 001 in the output, but it isn't. Transform your labels to their sine and cosine components. Transform back to angles from predictions.

And use both MAE and MSE as metrics. Which one you want as your loss function is up to your application. Gotta use that domain expertise for something.

Addendum: Your network is really, really deep and your inputs don't have mean 0 and std 1. You really need to either fix your normalization or throw a BatchNormalization layer in after Flatten. And you need to change your weight initialization to he_uniform from glorot. Also, your network is probably too deep.