0
$\begingroup$

I was asked by my supervisor to replicate a result from a former graduate student. My supervisor believes the result of that paper is not accurate and he asked me to find out why! The paper was about conducting a random forest classifier to classify some sort of diseases. I was reading through the simulation process, and I realize that the training set and the test test were generated separately (the same set of parameters were used in the simulations). In other word, the training was simulated first, then another simulation was carried on to generate the test set. My understanding for training and test set is to simulate one data set then use the same data for training and test set.

My question, is it correct to simulate the training and test set separately? Does it affect the accuracy of the classifier? any reference will be truly helpful.

$\endgroup$
0
$\begingroup$

In general, generating independently training and test sets is a legitimate option. The crucial aspect is that both the generating processes are equal. You can check this looking at this example from the author of the caret R package and the Applied Predictive Modeling book. However, it is something that can be easily proven with simulations. In what follows, both generating independently training and testing data or splitting the same data in training and testing subsets give the same results. The glm has a median accuracy of 92%.

# simulations with training and test data generating at the same time
n <- 100
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n)           # generate X
  z <- 1 + 4*x + rnorm(n) # linear combination with error
  pr <- 1/(1+exp(-z))     # inv-logit function
  y <- pr > 0.5           # 1 (True) if probability > 0.5
  df <-  data.frame(y = y, x = x) 
  train <- sample(x = 1:n, size=n%/%2, replace = F) # sampling training data units
  glm.fit <- glm(y ~ x, data = df[train,]) # fit on the training data
  predicted <- predict.glm(glm.fit, newdata = df[-train,]) # predict on the other data units
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df[-train,]$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy

# simulations with training and test data generating independently
n <- 100 # dataset size
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n%/%2)           # generate X
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.train <-  data.frame(y = y, x = x)
  glm.fit <- glm(y ~ x, data = df.train) # fit on the training data
  # generating independent test data
  x <- rnorm(n%/%2)
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.test <-  data.frame(y = y, x = x)
  predicted <- predict.glm(glm.fit, newdata = df.test) # predict on the test data
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df.test$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy
$\endgroup$
1
  • $\begingroup$ Thank you! It is helpful. I still do not see why or how it is possible looking from stochastic errors prospective. I am still looking for more clarification. $\endgroup$ Nov 22 '18 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.