1
$\begingroup$

I'm building a model to predict the flight delay. My dataset contains the following columns:

FL_DATE (contains months(1-12)), OP_CARRIER (One hot encoded data of Carrier names), ORIGIN(One hot encoded data of Origin Airport), Dest(one-hot encoded data of Dest Airport), CRS_DEP_TIME(Intended time of departure ex: 1015), DEP_TIME(Actual time of departure ex: 1017),DEP_DELAY(the difference between crs-dep ex: -2), ARR_DELAY(arrival delay ex: -2)

Screenshot of first 10 rows of my dataset

My target variable is ARR_DELAY. After checking my data, I have decided it is a regression problem. However, I'm not sure what method do I need to use for selecting the appropriate columns. On the other hand, I was plotting each column with ARR_DELAY to check their relation and got something like this: FL_TIME vs ARR_DELAY enter image description here

In such scenarios, if I have to build a model for such data which regression technique should I use?

PS: I'm new to Machine Learning. Please correct me If I'm heading in the wrong direction

$\endgroup$
1
  • $\begingroup$ It seems your question is really, 'How to choose the best features?' or 'How does one carry out feature selection?' When I consider a model I think what best describes the data, e.g. is it a linear model, square or KNN model? Your model is your estimate for the 'REAL' but unknown function. It seems you are looking to discover which variables are most important. I know you mention using python but I really like Kuhn's book on caret-R for its overall hands-on discussions. Try: topepo.github.io/caret/feature-selection-overview.html $\endgroup$
    – mccurcio
    Apr 29, 2020 at 1:27

3 Answers 3

0
$\begingroup$

One of my favorite tools for feature selection is the Random Forest. Consider giving it a try:
https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html

Random Forests are generally classifiers but they have the added advantage of being able to find and measure the variable's importance. It can be a long process but helpful. By using 'Gini importance' as a measure, RF can provide a bar chart that gives a relative comparison of the features with respect to which feature is best at separating signal from the noise. Check out this further discussion:
https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#workings

$\endgroup$
0
$\begingroup$

I think a better option will be that you try different regression techniques (it will be good for your learning curve) and see advantages and disadvantage.

For subset selection purposes few options are:

1- Best subset selection : in which you start with a null model and fill all possible combination of models and select best model on the basis of AIC, BIC or adjusted R2.

2-Forward selection: You start with a null model as well and add feature one by one to see which one gives lowest RSS or highest R2. Then add another feature again until you have the best model.

3-Backward selection: you start with a model with all p predictors remove one predictor evaluate adjusted R2 or AIC, BIC. do it again and again until you have a satisfactory model.

You don't have to do it manually both R and python have library that will help you in this.

$\endgroup$
1
  • $\begingroup$ I think you are getting feature selection confused with model selection. It seems obvious to me that the questioner is confused too. The main header is 'How to choose the best model' Not 'How to choose the best predictors or features'. Confusing the matter even more AIC and BIC are also used for feature selection and model selection. Choosing the best model requires Precision and Recall of your model. How accurate and specific, etc it is compared to others. en.wikipedia.org/wiki/Precision_and_recall $\endgroup$
    – mccurcio
    Apr 29, 2020 at 1:09
0
$\begingroup$

Try lazy regressor, it easily gives the performance of a number of different regressor models: https://pypi.org/project/lazypredict/

from lazypredict.Supervised import LazyRegressor
from sklearn import datasets
from sklearn.utils import shuffle
import numpy as np

boston = datasets.load_boston()
X, y = shuffle(boston.data, boston.target, random_state=13)
X = X.astype(np.float32)

offset = int(X.shape[0] * 0.9)

X_train, y_train = X[:offset], y[:offset]
X_test, y_test = X[offset:], y[offset:]

reg = LazyRegressor(verbose=0, ignore_warnings=False, custom_metric=None)
models, predictions = reg.fit(X_train, X_test, y_train, y_test)

print(models)


| Model                         | Adjusted R-Squared | R-Squared |  RMSE | Time Taken |
|:------------------------------|-------------------:|----------:|------:|-----------:|
| SVR                           |               0.83 |      0.88 |  2.62 |       0.01 |
| BaggingRegressor              |               0.83 |      0.88 |  2.63 |       0.03 |
| NuSVR                         |               0.82 |      0.86 |  2.76 |       0.03 |
| RandomForestRegressor         |               0.81 |      0.86 |  2.78 |       0.21 |
| XGBRegressor                  |               0.81 |      0.86 |  2.79 |       0.06 |
| GradientBoostingRegressor     |               0.81 |      0.86 |  2.84 |       0.11 |
| ExtraTreesRegressor           |               0.79 |      0.84 |  2.98 |       0.12 |
| AdaBoostRegressor             |               0.78 |      0.83 |  3.04 |       0.07 |
| HistGradientBoostingRegressor |               0.77 |      0.83 |  3.06 |       0.17 |
| PoissonRegressor              |               0.77 |      0.83 |  3.11 |       0.01 |
| LGBMRegressor                 |               0.77 |      0.83 |  3.11 |       0.07 |
| KNeighborsRegressor           |               0.77 |      0.83 |  3.12 |       0.01 |
| DecisionTreeRegressor         |               0.65 |      0.74 |  3.79 |       0.01 |
| MLPRegressor                  |               0.65 |      0.74 |  3.80 |       1.63 |
| HuberRegressor                |               0.64 |      0.74 |  3.84 |       0.01 |
| GammaRegressor                |               0.64 |      0.73 |  3.88 |       0.01 |
| LinearSVR                     |               0.62 |      0.72 |  3.96 |       0.01 |
| RidgeCV                       |               0.62 |      0.72 |  3.97 |       0.01 |
| BayesianRidge                 |               0.62 |      0.72 |  3.97 |       0.01 |
| Ridge                         |               0.62 |      0.72 |  3.97 |       0.01 |
| TransformedTargetRegressor    |               0.62 |      0.72 |  3.97 |       0.01 |
| LinearRegression              |               0.62 |      0.72 |  3.97 |       0.01 |
| ElasticNetCV                  |               0.62 |      0.72 |  3.98 |       0.04 |
| LassoCV                       |               0.62 |      0.72 |  3.98 |       0.06 |
| LassoLarsIC                   |               0.62 |      0.72 |  3.98 |       0.01 |
| LassoLarsCV                   |               0.62 |      0.72 |  3.98 |       0.02 |
| Lars                          |               0.61 |      0.72 |  3.99 |       0.01 |
| LarsCV                        |               0.61 |      0.71 |  4.02 |       0.04 |
| SGDRegressor                  |               0.60 |      0.70 |  4.07 |       0.01 |
| TweedieRegressor              |               0.59 |      0.70 |  4.12 |       0.01 |
| GeneralizedLinearRegressor    |               0.59 |      0.70 |  4.12 |       0.01 |
| ElasticNet                    |               0.58 |      0.69 |  4.16 |       0.01 |
| Lasso                         |               0.54 |      0.66 |  4.35 |       0.02 |
| RANSACRegressor               |               0.53 |      0.65 |  4.41 |       0.04 |
| OrthogonalMatchingPursuitCV   |               0.45 |      0.59 |  4.78 |       0.02 |
| PassiveAggressiveRegressor    |               0.37 |      0.54 |  5.09 |       0.01 |
| GaussianProcessRegressor      |               0.23 |      0.43 |  5.65 |       0.03 |
| OrthogonalMatchingPursuit     |               0.16 |      0.38 |  5.89 |       0.01 |
| ExtraTreeRegressor            |               0.08 |      0.32 |  6.17 |       0.01 |
| DummyRegressor                |              -0.38 |     -0.02 |  7.56 |       0.01 |
| LassoLars                     |              -0.38 |     -0.02 |  7.56 |       0.01 |
| KernelRidge                   |             -11.50 |     -8.25 | 22.74 |       0.01 |```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.