# Comparing training and validation data set Root MSE for a best subset regression?

I’ve a model with 14 dependent variables (all of them are significant) and 678 observations. I used best subset regression and validation set (33% of data for the validation) to find which statistical model has the lowest MSE (for my curiosity). I got the following graph which surprisingly the MSE for validation data set is always lower than training data set for all the models (from 1 to 14 dependent variables). Here is the code that I used,

library(MASS)
set.seed(1)
train=sample(seq(678),452,replace=FALSE)
train

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

val.errors=rep(NA,14)

x.test=model.matrix(HPV~.-Model.Types..code.-Year..code.,data=Mydata[-train,])
for(i in 1:14){
coefi=coef(regfit.exh,id=i)
pred=x.test[,names(coefi)]%*%coefi
val.errors[i]=mean((Mydata$HPV[-train]-pred)^2) } plot(sqrt(val.errors),ylab="Root MSE",ylim=c(3,12), pch=11, type="b") points(sqrt(regfit.exh$rss[-1]/452),col="blue",pch=11,type="b")
legend("topright",legend=c("Training","Validation"),col=c("blue","black"),pch=11)


How come the validation root MSE could always beat the training?. Any feedback would be appreciated.

• So, what is the question here? – Dawny33 Feb 25 '16 at 2:08
• @ Dawny, I'm thinking something is wrong. How come the validation root MSE could always beat the training? – Amir Feb 25 '16 at 2:22
• That's a general comment since I'm not sure what you have done but I do agree that looks odd.It could happpen for some data partitions. Repeat it changing the seed several times and check those results. In case it remains I would review the entire code. Good luck! – Rafael Muñoz-Mas Feb 27 '16 at 6:34
• Thanks Rafael. I changed the seed point but it didn't change a lot. – Amir Feb 29 '16 at 4:28

Im a little late, but better late than never. It looks like your line where you find the coefficients:

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")


should have data=Mydata[train,] instead of data=Mydata.