Training and Test set

Question

I was asked by my supervisor to replicate a result from a former graduate student. My supervisor believes the result of that paper is not accurate and he asked me to find out why! The paper was about conducting a random forest classifier to classify some sort of diseases. I was reading through the simulation process, and I realize that the training set and the test test were generated separately (the same set of parameters were used in the simulations). In other word, the training was simulated first, then another simulation was carried on to generate the test set. My understanding for training and test set is to simulate one data set then use the same data for training and test set.

My question, is it correct to simulate the training and test set separately? Does it affect the accuracy of the classifier? any reference will be truly helpful.

Answer 1

In general, generating independently training and test sets is a legitimate option. The crucial aspect is that both the generating processes are equal. You can check this looking at this example from the author of the caret R package and the Applied Predictive Modeling book. However, it is something that can be easily proven with simulations. In what follows, both generating independently training and testing data or splitting the same data in training and testing subsets give the same results. The glm has a median accuracy of 92%.

# simulations with training and test data generating at the same time
n <- 100
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n)           # generate X
  z <- 1 + 4*x + rnorm(n) # linear combination with error
  pr <- 1/(1+exp(-z))     # inv-logit function
  y <- pr > 0.5           # 1 (True) if probability > 0.5
  df <-  data.frame(y = y, x = x) 
  train <- sample(x = 1:n, size=n%/%2, replace = F) # sampling training data units
  glm.fit <- glm(y ~ x, data = df[train,]) # fit on the training data
  predicted <- predict.glm(glm.fit, newdata = df[-train,]) # predict on the other data units
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df[-train,]$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy

# simulations with training and test data generating independently
n <- 100 # dataset size
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n%/%2)           # generate X
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.train <-  data.frame(y = y, x = x)
  glm.fit <- glm(y ~ x, data = df.train) # fit on the training data
  # generating independent test data
  x <- rnorm(n%/%2)
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.test <-  data.frame(y = y, x = x)
  predicted <- predict.glm(glm.fit, newdata = df.test) # predict on the test data
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df.test$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy