How do I know if my model's result is good enough?

Question

I have a dataset of different people with their insurance cost. I have trained a neural-network to predict the insurance cost (charges column) based on the other features (age,bmi, etc.). Here is how my nn works:

insurance_model.compile(
    loss=tf.keras.losses.mae,
    optimizer=tf.keras.optimizers.Adamax(learning_rate=0.001),
    metrics=["mae"]
)

Here are some of the statistics of my target feature (charges column)

and here is the histogram of the values of charges column:

I evaluate my neural-network based on mae (mean-absolute-error) and at the end my mae=1300 for both train and test datasets.

So, I am wondering, if 1300 is good enough ?

How can i know, if any result is good enough ?

Answer 1

Mean absolute error is simple to interpret: in average, the value predicted by your model is off by 1300.

• First it looks good with respect to the range of the target value: 1300 is quite small for a range from 0 to 60000.
• The mean of the target value is 13k, so the error is 10% of the mean, this is fairly reasonable. If we extrapolate this could mean that the error is around 10% in average.

I'd suggest looking at the distribution of the errors (in particular the variance), in order to have some indication such as: "the system is x% likely to predict a value within y% of the correct value".

It could also be useful to study in which cases the largest errors take place. In particular does the error correlate with the value? This could matter because an error of 3k is huge if the target value is 2k, but it's small if the target value is 50k.

How can i know, if any result is good enough ?

By comparing to a baseline system. The baseline can be some previously existing system, a simplistic model (see example below), or some advanced model using a different method.

In this case the easiest baseline would be a system which always predicts the median. Calculating the MAE in this case gives some insight about how much your real system improves over a basic method.

You could also implement a simpler regression model, for instance with SVM or decision tree.