When exploring different techniques in machine learning (neural networks), I like to use binary classification problems as a test-bed, because it's very easy to understand how well the technique is working: 50% training/test accuracy is no better than chance, 70% is okay, 98% is very good, etc.
However, sometimes I need to use regression problems instead, and here I struggle to interpret my results. If I draw a scatter plot of training and test mean squared error before and after training, and before training I get around 0.1 loss, and afterwards I get 0.01 loss... is that... good? How do I recognize the difference between success, and barely doing any better than chance?
Typically I'm working with synthetic problems, by the way: fitting polynomials or other kinds of mathematical functions on random vector inputs.