I am trying out an R exercise where I have to randomly split my data into train/test set 50 times. On each iteration, I use lm() to train polynomial models up to 6th degree, and then predict the models on test set. I then have to do the following:

  1. Plot the test predictions of the polynomial models across all 50 iterations separately. (This I have done; see image below).
  2. Create a plot (either in the same plot or separately) to highlight the best model's test error across all 50 iterations. The error lines of the other models can be shaded in a lighter colour.

Part (2) is the one that confuses me. I am not sure what it is asking for. I thought about highlighting, for each polynomial, the line which returned the lowest test MSE out of the 50 iterations. However, I am quite sure this is wrong as I don't think "test predictions" mean the same as "error lines".

plot of test preds, 50 individual prediction lines per model

  • $\begingroup$ I'm not sure but I can think of 2 things: first, I'm surprised that all your graphs appear to be made of the same points: are my eyes wrong or do you always use the same test set? I think the test sets should be different. Second, my guess about point 2 would be that what is required is to plot the actual MSE value for the diferent iterations. it might also mean plotting the individual error for every point, but I don't see exactly what that would mean. $\endgroup$
    – Erwan
    Commented Sep 5, 2020 at 10:02
  • $\begingroup$ Oh yes I did randomly sample 50 test sets and ran all 6 models on each. But I superimposed the resulting 50 lines per model on a scatter of the full sample. This felt like the most sensible presentation to me. For point 2, would that refer to a graph with iterations as X and MSE as Y? And six lines with 50 points each. The second part does seem quite confusingly worded (no idea how an MSE graph can be in the same plots as the best fits), though my instructor insists there is nothing wrong with it. $\endgroup$
    – fluent
    Commented Sep 5, 2020 at 10:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.