2
$\begingroup$

I’ve a model with 14 dependent variables (all of them are significant) and 678 observations. I used best subset regression and validation set (33% of data for the validation) to find which statistical model has the lowest MSE (for my curiosity). I got the following graph which surprisingly the MSE for validation data set is always lower than training data set for all the models (from 1 to 14 dependent variables). Here is the code that I used,

library(MASS)
set.seed(1)
train=sample(seq(678),452,replace=FALSE)
train

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

val.errors=rep(NA,14)

x.test=model.matrix(HPV~.-Model.Types..code.-Year..code.,data=Mydata[-train,])     
for(i in 1:14){
  coefi=coef(regfit.exh,id=i)
  pred=x.test[,names(coefi)]%*%coefi
  val.errors[i]=mean((Mydata$HPV[-train]-pred)^2)
}

plot(sqrt(val.errors),ylab="Root MSE",ylim=c(3,12), pch=11, type="b")
points(sqrt(regfit.exh$rss[-1]/452),col="blue",pch=11,type="b")
legend("topright",legend=c("Training","Validation"),col=c("blue","black"),pch=11)

enter image description here

How come the validation root MSE could always beat the training?. Any feedback would be appreciated.

Improve this question
$\endgroup$
4
  • $\begingroup$ So, what is the question here? $\endgroup$
    – Dawny33
    Commented Feb 25, 2016 at 2:08
  • $\begingroup$ @ Dawny, I'm thinking something is wrong. How come the validation root MSE could always beat the training? $\endgroup$
    – Amir
    Commented Feb 25, 2016 at 2:22
  • $\begingroup$ That's a general comment since I'm not sure what you have done but I do agree that looks odd.It could happpen for some data partitions. Repeat it changing the seed several times and check those results. In case it remains I would review the entire code. Good luck! $\endgroup$
    – Rafael Muñoz-Mas
    Commented Feb 27, 2016 at 6:34
  • $\begingroup$ Thanks Rafael. I changed the seed point but it didn't change a lot. $\endgroup$
    – Amir
    Commented Feb 29, 2016 at 4:28

1 Answer 1

Reset to default
1
$\begingroup$

Im a little late, but better late than never. It looks like your line where you find the coefficients: 

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

should have data=Mydata[train,] instead of data=Mydata.

Your model had already seen your test samples, so the validation error is not an accurate assessment.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your caught, you were right. I updated the code. However, the results are better but I don't have a clear explanations for that. Here is the updated graph. ![enter image description here](i.sstatic.net/nPT1U.png) Thank you again for your insights and feedback. $\endgroup$
    – Amir
    Commented Apr 30, 2016 at 3:07
  • $\begingroup$ if you continue on the x axis "Index" for another 20 variables, do you start seeing the error increase? The fact that the validation error flattens out like that when you presumably add too many variables to the model is as suspicious as the lower validation set MSE. $\endgroup$
    – TBSRounder
    Commented May 1, 2016 at 17:52
  • $\begingroup$ @ TBSRounder, I do hear you and thanks for the comment. However, the contribution of some of the variables are so low but the coefficients' sign that found were making sense and they were significant also. Do you think I need to remove those; however, they were meaningful and significant? Thank you. $\endgroup$
    – Amir
    Commented May 5, 2016 at 17:35
  • $\begingroup$ No Problem! It depends on your goals. If you're looking for best prediction with some storytelling, I would leave them in as long as your validation error gets better. If you're planning on inference, I would take a close look at the variables and decide what to leave in/out based on the context and multiple testing constraints. Are these variables in your pool for subset selection already trimmed down from a bunch of variables? Are you saying when you have 14 variables in your model that all of them are significant? $\endgroup$
    – TBSRounder
    Commented May 6, 2016 at 12:23
  • $\begingroup$ Thank you TBSRounder. In fact there are 13 variables which 2 of them are categorical ones at 3 levels (in total 11+2+2=15 variables that is shown in the Figure). However, the significant level is different for various variables but all variables are significant at the level of 10%. Thanks again for your insights and feedback. $\endgroup$
    – Amir
    Commented May 6, 2016 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.