How to run a BE or FS Stepwise Regression on each dataset in a file folder full of datasets using lapply or map (without a loop)

Question

All of the code in this question can be found in my GitHub Repository for this research project on Estimated Exhaustive Regression. Specifically, in the "Both BE & FS script" and "LASSO code" Rscripts, and you may use the significantly truncated file folder of datasets "sample_obs(20)" rather than "spencer" because the former only contains 20 csvs while the latter contains 58.5k!

After loading in and sorting the N datasets, I ran my N LASSO Regressions, with one for each dataset using lapply in the following manner:

set.seed(11)     # to ensure replicability
LASSO_fits <- lapply(datasets, function(i) 
               enet(x = as.matrix(select(i, starts_with("X"))), 
               y = i$Y, lambda = 0, normalize = FALSE))


# This stores and prints out all of the regression 
# equation specifications selected by LASSO when called
set.seed(11)     # to ensure replicability
LASSO_Coeffs <- lapply(LASSO_fits, 
                       function(i) predict(i, x = as.matrix(select(i, starts_with("X"))), 
                                           s = 0.1, mode = "fraction", 
                                           type = "coefficients")[["coefficients"]])

Now, I want to replicate the above sort of code but for both BE and FS Stepwise Regression as well using the step function from the stats package and avoiding the use of any loops at all costs!

I will be running all three of these Benchmark Variable Selection Algorithms on 260k synthetic datasets, so the increase in runtime from using a loop would make running my code practically infeasible.

p.s. The datasets are loaded in and sorted via the following code:

directory_paths <- "~/GMU folders(local)/DAEN_698/other datasets/sample obs(20 csvs)"
filepaths_list <- list.files(path = directory_paths, full.names = TRUE, recursive = TRUE)
length(filepaths_list)
str(filepaths_list)


# reformat the names of each of the csv file formatted datasets
DS_names_list <- basename(filepaths_list)
DS_names_list <- tools::file_path_sans_ext(DS_names_list)
head(DS_names_list, n = 4)

my_order = DS_names_list |> 
  # split apart the numbers
  strsplit(split = "-", fixed = TRUE) |>
  unlist() |> 
  # convert them to numeric and get them in a data frame
  as.numeric() |> 
  matrix(nrow = length(DS_names_list), byrow = TRUE) |>
  as.data.frame() |>
  # get the appropriate ordering to sort the data frame
  do.call(order, args = _)

DS_names_list = DS_names_list[my_order]

filepaths_list = filepaths_list[my_order]

## This line reads all of the data in each of the csv files 
## using the name of each store in the list we just created.
datasets <- lapply(filepaths_list, read.csv)

Answer 1

First, on that many datasets, consider using fread from the data.table package rather than the standard but slow read.csv.

As for the stepwise regression function, using the step function from the stats package to do it can do done in this manner for a Forward Stepwise Regression:

FS_fits <- lapply(X = datasets, \(X) {
  nulls <- lm(X$Y ~ 1, data = X)
  full_models <- lm(X$Y ~ ., X)
  forward <- stats::step(object = nulls, direction = 'forward',
                         scope = formula(full_models), trace = FALSE)}) )

Something quite similar should do for Backward Stepwise as well, except you won't need a nulls regression in it, the object will be set equal to full_models, and the direction will be 'backward' of course.