2
$\begingroup$

I’ve a model with 14 dependent variables (all of them are significant) and 678 observations. I used best subset regression and validation set (33% of data for the validation) to find which statistical model has the lowest MSE (for my curiosity). I got the following graph which surprisingly the MSE for validation data set is always lower than training data set for all the models (from 1 to 14 dependent variables). Here is the code that I used,

library(MASS)
set.seed(1)
train=sample(seq(678),452,replace=FALSE)
train

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

val.errors=rep(NA,14)

x.test=model.matrix(HPV~.-Model.Types..code.-Year..code.,data=Mydata[-train,])     
for(i in 1:14){
  coefi=coef(regfit.exh,id=i)
  pred=x.test[,names(coefi)]%*%coefi
  val.errors[i]=mean((Mydata$HPV[-train]-pred)^2)
}

plot(sqrt(val.errors),ylab="Root MSE",ylim=c(3,12), pch=11, type="b")
points(sqrt(regfit.exh$rss[-1]/452),col="blue",pch=11,type="b")
legend("topright",legend=c("Training","Validation"),col=c("blue","black"),pch=11)

enter image description here

How come the validation root MSE could always beat the training?. Any feedback would be appreciated.

$\endgroup$
4
  • $\begingroup$ So, what is the question here? $\endgroup$ – Dawny33 Feb 25 '16 at 2:08
  • $\begingroup$ @ Dawny, I'm thinking something is wrong. How come the validation root MSE could always beat the training? $\endgroup$ – Amir Feb 25 '16 at 2:22
  • $\begingroup$ That's a general comment since I'm not sure what you have done but I do agree that looks odd.It could happpen for some data partitions. Repeat it changing the seed several times and check those results. In case it remains I would review the entire code. Good luck! $\endgroup$ – Rafael Muñoz-Mas Feb 27 '16 at 6:34
  • $\begingroup$ Thanks Rafael. I changed the seed point but it didn't change a lot. $\endgroup$ – Amir Feb 29 '16 at 4:28
1
$\begingroup$

Im a little late, but better late than never. It looks like your line where you find the coefficients:

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

should have data=Mydata[train,] instead of data=Mydata.

Your model had already seen your test samples, so the validation error is not an accurate assessment.

$\endgroup$
7
  • $\begingroup$ Thanks for your caught, you were right. I updated the code. However, the results are better but I don't have a clear explanations for that. Here is the updated graph. ![enter image description here](i.stack.imgur.com/nPT1U.png) Thank you again for your insights and feedback. $\endgroup$ – Amir Apr 30 '16 at 3:07
  • $\begingroup$ if you continue on the x axis "Index" for another 20 variables, do you start seeing the error increase? The fact that the validation error flattens out like that when you presumably add too many variables to the model is as suspicious as the lower validation set MSE. $\endgroup$ – TBSRounder May 1 '16 at 17:52
  • $\begingroup$ @ TBSRounder, I do hear you and thanks for the comment. However, the contribution of some of the variables are so low but the coefficients' sign that found were making sense and they were significant also. Do you think I need to remove those; however, they were meaningful and significant? Thank you. $\endgroup$ – Amir May 5 '16 at 17:35
  • $\begingroup$ No Problem! It depends on your goals. If you're looking for best prediction with some storytelling, I would leave them in as long as your validation error gets better. If you're planning on inference, I would take a close look at the variables and decide what to leave in/out based on the context and multiple testing constraints. Are these variables in your pool for subset selection already trimmed down from a bunch of variables? Are you saying when you have 14 variables in your model that all of them are significant? $\endgroup$ – TBSRounder May 6 '16 at 12:23
  • $\begingroup$ Thank you TBSRounder. In fact there are 13 variables which 2 of them are categorical ones at 3 levels (in total 11+2+2=15 variables that is shown in the Figure). However, the significant level is different for various variables but all variables are significant at the level of 10%. Thanks again for your insights and feedback. $\endgroup$ – Amir May 6 '16 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.