How close is close enough, with regression?

When exploring different techniques in machine learning (neural networks), I like to use binary classification problems as a test-bed, because it's very easy to understand how well the technique is working: 50% training/test accuracy is no better than chance, 70% is okay, 98% is very good, etc.

However, sometimes I need to use regression problems instead, and here I struggle to interpret my results. If I draw a scatter plot of training and test mean squared error before and after training, and before training I get around 0.1 loss, and afterwards I get 0.01 loss... is that... good? How do I recognize the difference between success, and barely doing any better than chance?

Typically I'm working with synthetic problems, by the way: fitting polynomials or other kinds of mathematical functions on random vector inputs.

• Are you sure $98\%$ classification accuracy is always very good? Granted, you said binary classification, but $98\%$ accuracy is rather pedestrian for MNIST digit classification. // I dispute the notion that there are absolute measures of performance that match up with letter grades in school. It depends on the context and the problem. Not every problem will have the same tolerance for errors. It would be convenient if simple measures could replace critical thinking, but we have no such luck.
– Dave
Commented Oct 24, 2021 at 15:50
• Additionally, classification accuracy is a surprisingly problematic performance (1) (2) (3).
– Dave
Commented Oct 24, 2021 at 15:55
• Accuracy works for you because you intuitively understand it. If you want to have this kind of understanding of your error for regression use a metric that makes intuitive sense to you. The easiest ones to grok is probably mean absolute error. Commented Oct 24, 2022 at 20:20