# Training and Test set

I was asked by my supervisor to replicate a result from a former graduate student. My supervisor believes the result of that paper is not accurate and he asked me to find out why! The paper was about conducting a random forest classifier to classify some sort of diseases. I was reading through the simulation process, and I realize that the training set and the test test were generated separately (the same set of parameters were used in the simulations). In other word, the training was simulated first, then another simulation was carried on to generate the test set. My understanding for training and test set is to simulate one data set then use the same data for training and test set.

My question, is it correct to simulate the training and test set separately? Does it affect the accuracy of the classifier? any reference will be truly helpful.

In general, generating independently training and test sets is a legitimate option. The crucial aspect is that both the generating processes are equal. You can check this looking at this example from the author of the caret R package and the Applied Predictive Modeling book. However, it is something that can be easily proven with simulations. In what follows, both generating independently training and testing data or splitting the same data in training and testing subsets give the same results. The glm has a median accuracy of 92%.

# simulations with training and test data generating at the same time
n <- 100
accuracy <- vector("numeric")
for (i in 1:1000){
#create data
x <- rnorm(n)           # generate X
z <- 1 + 4*x + rnorm(n) # linear combination with error
pr <- 1/(1+exp(-z))     # inv-logit function
y <- pr > 0.5           # 1 (True) if probability > 0.5
df <-  data.frame(y = y, x = x)
train <- sample(x = 1:n, size=n%/%2, replace = F) # sampling training data units
glm.fit <- glm(y ~ x, data = df[train,]) # fit on the training data
predicted <- predict.glm(glm.fit, newdata = df[-train,]) # predict on the other data units
accuracy=c(accuracy, sum(diag(table(predicted>0.5, df[-train,]$y)))/(n%/%2)) # collect accuracy } quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy # simulations with training and test data generating independently n <- 100 # dataset size accuracy <- vector("numeric") for (i in 1:1000){ #create data x <- rnorm(n%/%2) # generate X z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error pr <- 1/(1+exp(-z)) # inv-logit function y <- pr > 0.5 # 1 (True) if probability > 0.5 df.train <- data.frame(y = y, x = x) glm.fit <- glm(y ~ x, data = df.train) # fit on the training data # generating independent test data x <- rnorm(n%/%2) z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error pr <- 1/(1+exp(-z)) # inv-logit function y <- pr > 0.5 # 1 (True) if probability > 0.5 df.test <- data.frame(y = y, x = x) predicted <- predict.glm(glm.fit, newdata = df.test) # predict on the test data accuracy=c(accuracy, sum(diag(table(predicted>0.5, df.test$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy

• Thank you! It is helpful. I still do not see why or how it is possible looking from stochastic errors prospective. I am still looking for more clarification. Nov 22 '18 at 16:06