Which measurement(s) should one choose to compare two regression models?
After modifying a learning algorithm(specifically, a regression algorithm, let's call it M1) to generate another learning algorithm M2, how to validate if the above modification is efficient?
here is what I did(with 10-fold cross-validation)
I choose MSE as the only measurement, at each run, for M1 and M2, calculate the MSE of both the training and testing set.
And the result shows that:
- average MSE of the training set of 10 runs: M2 < M1
- average MSE of the testing set of 10 runs: M2 < M1
according to the above list, can we draw a conclusion that M2 is better than M1? thus, the modification of algorithm M1 is efficient(at least on this dataset)?
Did I miss some other important measurements? Is there a rule of thumb of comparing two regression models?