1
$\begingroup$

When exploring different techniques in machine learning (neural networks), I like to use binary classification problems as a test-bed, because it's very easy to understand how well the technique is working: 50% training/test accuracy is no better than chance, 70% is okay, 98% is very good, etc.

However, sometimes I need to use regression problems instead, and here I struggle to interpret my results. If I draw a scatter plot of training and test mean squared error before and after training, and before training I get around 0.1 loss, and afterwards I get 0.01 loss... is that... good? How do I recognize the difference between success, and barely doing any better than chance?

Typically I'm working with synthetic problems, by the way: fitting polynomials or other kinds of mathematical functions on random vector inputs.

Improve this question
$\endgroup$
3
  • $\begingroup$ Are you sure $98\%$ classification accuracy is always very good? Granted, you said binary classification, but $98\%$ accuracy is rather pedestrian for MNIST digit classification. // I dispute the notion that there are absolute measures of performance that match up with letter grades in school. It depends on the context and the problem. Not every problem will have the same tolerance for errors. It would be convenient if simple measures could replace critical thinking, but we have no such luck. $\endgroup$
    – Dave
    Oct 24, 2021 at 15:50
  • $\begingroup$ Additionally, classification accuracy is a surprisingly problematic performance (1) (2) (3). $\endgroup$
    – Dave
    Oct 24, 2021 at 15:55
  • $\begingroup$ Accuracy works for you because you intuitively understand it. If you want to have this kind of understanding of your error for regression use a metric that makes intuitive sense to you. The easiest ones to grok is probably mean absolute error. $\endgroup$
    – Michael Higgins
    Oct 24, 2022 at 20:20

2 Answers 2

Reset to default
1
$\begingroup$

I think it is a far question, since we should always try to translate any metric result to a useful interpretation.

When talking about regression, we can judge the success of our model by checking whether that MAE or RMSE exceed what we are willing to accept.
For instance, if we want to predict the temperature for the next days, and our result (in this case in a time series problem type) is a MAE of about 2ºC (after re-scaling our data to the natural units like celsius degrees), is it enough for our goal? Is a mean (absolute) error of 2ºC above or below the real temperature enough for the case being studied? Maybe it is for an approximate model, but maybe it is not a good result if we want to use it in an environment where tempreature could have more impact on whatever being studied...

Another additional chance you have is estimating confidence intervals (see this answer) for your model coefficents and, more precisely, prediction intervals (see this other validated answer) to have an estimate for your prediction together with an interval.

Improve this answer
$\endgroup$
0
$\begingroup$

The quality of a regression model is domain specific. Different problems and data sets will have different standards for relative model improvement.

Typically machine learning models are measured by their performance on the hold-out dataset (i.e., validation or test dataset). Thus, the training loss values are not as important in measuring model quality.

One way to frame the interpretation is to pick several evaluation metrics, such as Mean Square Error(MSE), Root Mean Square Error(RMSE), or Mean Absolute Error(MAE), and see which ones are improving on a hold-out dataset.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.