# Methods of building machine learning models

I have seen pages where they mention 5 methods of building models.

1) All-in
2) Backward Elimination
3) Forward Selection
4) Bidirectional Elimination
5) Score Comparision


I usually implement a linear regression or any algorithm using sklearn

from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X, y)
y_pred = lr.predict(X_test)


How can one implement those 5 methods of building models?

Can anyone explain the importance of this and What is most commonly used one out of these?

Strictly speaking, those are not methods of "building" models, they are feature selection strategies.

Most of them are implemented in Scikit-learn.

As Qusai said in his answer, these are not methods to build models, these are selection mechanisms. There are others as well, like feature engineering (for instance dimensionality reduction can be seen as feature engineering).

Starting from your example, and assuming that X is your full data:

from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X, y)
y_pred = lr.predict(X_test)


This is the all-in. From this, backward elimination is removing one of the columns and keeping the n-1 with the best result.

Then forward is starting from 0 and adding 1 feature, so just one column at a time, comparing all potential ones.

The bidirectional is to try adding or removing at each step (because non linear models may sometimes be better without one feature, but better with two of them).

Each time, compare the score, and more precisely, compare the cross-validated score. In a way, only step 5 is what you ALWAYS do.

Backward Elimination

Step 1: Select a significance level to stay in the model (Eg: SL=0.05).

Step 2: Fit the full model with the possible predictors.

Step 3: Consider the predictor with the highest p-value. If P>SL, go to step 4 otherwise your model is completed.

Step 4: Remove the predictor

Step 5: Fit models without this variable and move to step 3.

Forward Selection

Step 1: Select a significance level to enter in the model (Eg: SL=0.05)

Step 2: Fit all Simple Regression models $$y$$ ~ $$X_n$$, select the one with the lowest p-value.

Step 3: Keep the variable and fit all possible models with one extra predictor added to the one(s) you already have.

Step 4: Consider the predictor with the lowest p-value. If p-value < SL, go to Step3, else the model is completed.

Bidirectional Elimination

Step 1: Select a significance level to stay in the model (Eg: SLENT=0.05, SLSTAY=0.05)

Step 2: Perform the next step of forward selection (new variables must have p-value<SLENTT to enter)

Step 3: Perform all steps of backward elimination (old variables must have p-value<SLSTAY to stay) repeat step 2 and 3 until no new variables than move to step 5.

Step 4: If no new variables can enter and no variables can exit, so it is the final model.

All Possible Models

Step 1: Select the criterion of the goodness of fit.

Step 2: Construct all possible regression models $$2^{N-1}$$ total combinations.

Step 3: Select one with the best criterion