4
$\begingroup$

number of features: 12 , -15 < each feature < 15

number of targets: 6 , 0 < each target < 360

number of examples: 262144

my normalization: I normalized the features so that they are between 0 and 1. I normalized the targets so that they are between 1 and 10.

This is the model that I am using:

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(6, activation='linear')
])

model.compile(optimizer="rmsprop", loss='mean_squared_error', metrics=['accuracy'])

model.fit(training_x, training_y, epochs=10, batch_size=100)

This is the best result that I have got (training):

235929/235929 [==============================] - 8s 33us/step - loss: 8.9393e-04 - acc: 0.6436

testing:

loss: 0.00427692719418488

acc: 0.033187106618348276

I get almost 0% accuracy on the test set! I need a model to solve this ML problem.

$\endgroup$
1
  • $\begingroup$ Just try not normalizing the targets, and use another metric, like Mean Absolute Error ("mae") $\endgroup$
    – ignatius
    Apr 1, 2019 at 12:58

2 Answers 2

4
$\begingroup$

Accuracy is a metric for classification, not regression.

$$Accuracy = \frac{\text{Correct classification}}{\text{Number of classifications}}$$

So when you use accuracy for regression only the values where actual_label == predicted_label are evaluated as true are counted as correct classifications. That will happen quite rarely when you are doing regression, resulting in an accuracy that is close to zero.

Instead you should use something like mean absolute error or mean squared error as validation metrics for regression.

$\endgroup$
10
  • $\begingroup$ 235929/235929 [==============================] - 13s 54us/step - loss: 1.0431 - mean_absolute_error: 0.5507 $\endgroup$
    – Kasra
    Apr 1, 2019 at 13:25
  • $\begingroup$ You want mean absolute error to be as close to zero as possible. It is basically the average error in units that your model makes. 0.5507 might be a good score depending on the size of your labels. $\endgroup$ Apr 1, 2019 at 13:29
  • 1
    $\begingroup$ But is important to note that MAE will be affected by any normalization you perform on your label. So be sure to take that into consideration. $\endgroup$ Apr 1, 2019 at 13:30
  • $\begingroup$ On the test set, I got MAE of 1.4721086428362338. Is it acceptable? How can I improve performance? $\endgroup$
    – Kasra
    Apr 1, 2019 at 13:31
  • $\begingroup$ Depends on the size of your labels. If you for example are predicting house prices (big numbers) and on average is wrong with only 1.4721086428362338 dollars then your results are great. You can try running print(np.mean(training_y)) to get a sense of the size of your labels $\endgroup$ Apr 1, 2019 at 13:34
1
$\begingroup$

You don't have to normalize regression targets but in a different case you might have wanted to scale them so that the loss of one output doesn't dominate over the loss for other outputs.

The very important thing here is that you are predicting angles which are periodically bounded but you have them as continuous values. So 359 should be very close to 001 in the output, but it isn't. Transform your labels to their sine and cosine components. Transform back to angles from predictions.

And use both MAE and MSE as metrics. Which one you want as your loss function is up to your application. Gotta use that domain expertise for something.

Addendum: Your network is really, really deep and your inputs don't have mean 0 and std 1. You really need to either fix your normalization or throw a BatchNormalization layer in after Flatten. And you need to change your weight initialization to he_uniform from glorot. Also, your network is probably too deep.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.