0
$\begingroup$

I am currently doing research with a coauthor and collaborator comparing a new optimal model selection procedure he has proposed via Monte Carlo Simulation of the new procedure vs 2 benchmarks, LASSO & Backward Elimination Stepwise. In order to compare the % of factors correctly selected and the % of factors selected which are spurious by his new procedure vs LASSO vs (BE) Stepwise head-to-head, all we will be comparing are the regressors/"factors" selected by each of the 3 in terms the aforementioned criteria.

This is made not only possible, but extremely simple because when my collaborator created the randomly generated synthetic sample observations on which to run his procedure vs LASSO & BE, he did so in such a way that the correct number of factors in the true underlying population model is known for each of the 47,000 individual (500 by 31) datasets stored in their own csv files within the same file folder.

The main issue here is that I am still a novice when it comes to writing and running code in R unfortunately. So, I have already written the following code, all of which works/runs (besides the last 3 lines which is why I am asking this question here):

directory_path <- "~/DAEN_698/sample_obs"
file_list <- list.files(path = directory_path, full.names = TRUE, recursive = TRUE)
head(file_list, n = 2)
> head(file_list, n = 2)
[1] "C:/Users/Spencer/Documents/DAEN_698/sample_obs2/0-5-1-1.csv"
[2] "C:/Users/Spencer/Documents/DAEN_698/sample_obs2/0-5-1-2.csv"

# Create another list with the just the "n-n-n-n" part of the names of of each dataset
DS_name_list = stri_sub(file_list, 49, 55)
head(DS_name_list, n = 3)
> head(DS_name_list, n = 3)
[1] "0-5-1-1" "0-5-1-2" "0-5-1-3"

# This command reads all the data in each of the N csv files via their names 
# stored in the 'file_list' list of characters.
csvs <- lapply(file_list, read.csv)

### Run a Backward Elimination Stepwise Regression on each of the N csvs.
# Assign the full model (meaning the one with all 30 candidate regressors 
# included as the initial model in step 1).
# This is crucial because if the initial model has less than the number of 
# total candidate factors for Stepwise to select from in the datasets, 
# then it could miss 1 or more of the true factors. 
full_model <- lapply(csvs, function(i) {
  lm(formula = Y ~ ., data = i) })

# my failed attempt at figuring it out myself
set.seed(50)      # for reproducibility
BE_fits3 <- lapply(full_model, function(i) {step(object = i[["coefficients"]], 
direction = 'backward', scope = formula(full_model), trace = 0)})

When I hit run on the above 2 lines of code after setting the seed, I get the following error message in the Console:

Error in terms`(object) : object 'i' not found

Post Scrip/Mini Appendix: To briefly elaborate a bit further on why it is absolutely essential that the initial model when running a Backward Elimination version of Stepwise Regression, consider the following example: Let us say that we start out with an initial model of 25, so, X1:X26 instead of X1:X30, in that case, it would be possible to miss out on Stepwise Regression j being able to select/choose 1 or more of the IVs/factors from X26 through X30, especially if 1 or more of those really are included in the true underlying population model that characterizes dataset j.

$\endgroup$

1 Answer 1

0
$\begingroup$

If you are willing to deal with the runtime issues of going with a for-loop, try this:

full_model <- vector("list", length = length(csvs))
BE_fits <- vector("list", length = length(csvs))

This is to initialize everything before running the following loop:

for(i in seq_along(csvs)) {
  full_model[[i]] <- lm(formula = Y ~ ., data = csvs[[i]])
  BE_fits[[i]] <- step(object = full_model[[i]], 
                        scope = formula(full_model[[i]]),
                        direction = 'backward', trace = 0) }

It is important to remember to include the formula argument within the scope argument of the step function, it may throw you an error if you omit this detail.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.