When I use linear regression in machine learning, variables selection is same as choosing turning parameters?

Question

I am a newbie in machine learning. After days of studying the ideas of machine learning, I have made some conclusions, which are below (I only consider supervised learning).

Step 1: Data splitting Before we go to data processing, the dataset should be split into training and testing. The training dataset will go through the checking (validation) process while the testing dataset is kept unchanged to assess model performance.

Step 2: Cross-validation k-fold (a lot of methods but example)

Regarding applying traditional modelling (linear regression) to machine learning:

When we do cross-validation (k-fold) the aim is to choose the model with the best input variables (basing on AIC, BIC... ). This is because the linear regression model has no turning parameters for optimising (only variables). It is true that LASSO or PCA are not considered in this case because they have to do variable selection (feature selection) themselves. After that, the model with the best input variables will be used to check model performance (calculating mean absolute error (MAE), mean square error (MSE)...)

Regarding machine learning algorithms:

The aim of cross-validation (k-fold) is to choose the most proper turning parameters (ex: n_estimators, max_depth in random forest) After that, the model with the best turning parameters will be used to check model performance (calculating MAE, MSE...)

!!!Important: only the training dataset is used for cross-validation

Step 3: Assessing model performance

The MAE, MSE will be calculated basing on models with best input variables (linear regression) or turning parameters (machine learning algorithms). The testing dataset is used in this step.

This is all about steps for doing machine learning (ideas) due to my understanding. Thus, would it be right that variable selection for traditional modelling (linear regression) is similar to choosing the best parameters during the cross-validation process?

Besides, if I have made some mistakes in the content above, would you please show it to me?

Answer 1

Understanding Variable Selection in Linear Regression: Machine Learning vs. Traditional Statistics

Variable selection is a cornerstone of both machine learning and traditional statistical modelling. Although these fields share foundational techniques, their approaches differ due to distinct goals, assumptions, and practical considerations.

---

Mathematical Definitions

Linear regression models the relationship between a dependent variable $Y$ and independent variables $X_1, X_2, \dots, X_p$:

$$ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon $$

Where:

• $\beta_0, \beta_1, \dots, \beta_p$ are coefficients to estimate,
• $\epsilon$ is the random error term.

Key Differences:

• Traditional Statistics: Focuses on identifying statistically significant predictors, prioritising interpretability and adherence to assumptions like linearity, normality, and independence of errors.
• Machine Learning: Centres on predictive accuracy, relaxing assumptions and scaling methods for high-dimensional data.

---

Key Differences Between the Two Fields

1. Context and Goals

• Traditional Statistics: Prioritises interpretability. Variable selection focuses on identifying significant predictors using hypothesis testing (eg., p-values, confidence intervals).
• Machine Learning: Optimises for predictive performance on unseen data. Models are evaluated using metrics like RMSE and cross-validation accuracy.

2. Variable Selection Techniques

• Traditional Statistics:
  • Stepwise selection (forward, backward, bidirectional). However, it is important to note that these techniques are not recommended.
  • Regularisation methods, eg., Lasso (Tibshirani, 1996) and Ridge regression (Hoerl & Kennard, 1970).
• Machine Learning:
  • Feature importance from tree-based models (eg., Random Forest).
  • Embedded techniques in regularised models, eg., Elastic Net (Zou & Hastie, 2005).
  • Automation via pipelines (eg., sklearn's SelectFromModel).

3. Assumptions and Interpretability

• Traditional Statistics: Assumptions, eg., homoscedasticity and residual normality, guide validity.
• Machine Learning: Assumptions are relaxed, with a focus on integrating non-linear features or interactions.

---

Commonalities in Variable Selection

Despite differing goals, both fields share these principles:

1. Regularisation: Techniques like Lasso (L1) and Ridge (L2) reduce multicollinearity and prevent overfitting.
2. Model Simplicity: Parsimonious models are favoured for robustness and generalisability.
3. Evaluation Metrics: Metrics align with specific goals, eg., adjusted $R^2$ for explanatory models vs. accuracy metrics in machine learning.

---

Practical Considerations

1. Dataset Size: Machine learning methods handle large, high-dimensional datasets, requiring scalable selection techniques.
2. Interpretability vs. Predictive Power: Traditional models favour clarity, while machine learning prioritises predictive accuracy.
3. Workflow Automation: Machine learning tools (eg., AutoML) streamline variable selection and feature engineering.

---

Conclusion

Variable selection techniques align in principle but differ in execution between machine learning and traditional statistics. While statistics emphasises interpretability and theoretical validity, machine learning focuses on scalability and predictive accuracy. The choice of approach should be based on the goals of the analysis.

---

References

• Tibshirani, R. (1996). "Regression Shrinkage and Selection via the Lasso." Journal of the Royal Statistical Society: Series B (Methodological).
• Zou, H., & Hastie, T. (2005). "Regularisation and Variable Selection via the Elastic Net." Journal of the Royal Statistical Society: Series B.
• Hoerl, A. E., & Kennard, R. W. (1970). "Ridge Regression: Biased Estimation for Nonorthogonal Problems." Technometrics.