Skip to main content
Share Your Experience: Take the 2024 Developer Survey
63 votes
Accepted

Does XGBoost handle multicollinearity by itself?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
Sandeep S. Sandhu's user avatar
29 votes
Accepted

How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif

Intuition We have two classes and we want to find a score for each feature saying "how well this feature discriminates between two classes". Now look at the figure bellow. There are two ...
Kasra Manshaei's user avatar
28 votes

Does XGBoost handle multicollinearity by itself?

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
dalloliogm's user avatar
27 votes

Does XGBoost handle multicollinearity by itself?

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, ...
Jiaxiang's user avatar
  • 400
24 votes
Accepted

Feature selection vs Feature extraction. Which to use when?

Adding to The answer given by Toros, These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember) feature extraction and feature engineering: ...
Aditya's user avatar
  • 2,470
20 votes

What is difference between one hot encoding and leave one out encoding?

They are probably using "leave one out encoding" to refer to Owen Zhang's strategy. From here The encoded column is not a conventional dummy variable, but instead is the mean response over ...
Dex Groves's user avatar
20 votes
Accepted

Any "rules of thumb" on number of features versus number of instances? (small data sets)

Multiple papers have opined that only in rare cases is there a known distribution of the error as a function of the number of features and sample size. The error surface for a given set of instances, ...
shark8me's user avatar
  • 386
19 votes

Does scikit-learn have a forward selection/stepwise regression algorithm?

Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and ...
David Dale's user avatar
  • 1,551
18 votes
Accepted

In ML why selecting the best variables?

You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce ...
fractalnature's user avatar
17 votes
Accepted

Dissmissing features based on correlation with target variable

You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. ...
Sean Owen's user avatar
  • 6,595
17 votes

How to do stepwise regression using sklearn?

Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and ...
David Dale's user avatar
  • 1,551
15 votes
Accepted

How to determine feature importance in a neural network?

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. ...
scholle's user avatar
  • 174
15 votes
Accepted

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

Not every seed is the same. Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility: ...
Noah Weber's user avatar
  • 5,679
15 votes
Accepted

Why continuous features are more important than categorical features in decision tree models?

It could be the way that you encode categorical variables. If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less ...
Carlos Mougan's user avatar
13 votes

List of feature engineering techniques

Missing Data Imputation: Complete case analysis Mean / Median / Mode imputation Random Sample Imputation Replacement by Arbitrary Value Missing Value Indicator Multivariate imputation ...
Sole G's user avatar
  • 281
13 votes

When should I use StandardScaler and when MinMaxScaler?

StandardScaler and MinMaxScaler are more common when dealing with continuous numerical data. One possible preprocessing approach for OneHotEncoding scaling is "soft-binarizing" the dummy ...
UrbanoFonseca's user avatar
12 votes

Does scikit-learn have a forward selection/stepwise regression algorithm?

As of version 0.24, it does! Announcement, documentation
Ben Reiniger's user avatar
  • 11.9k
12 votes
Accepted

Is feature selection necessary?

Feature selection might be consider a stage to avoid. You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are ...
DaL's user avatar
  • 2,643
12 votes
Accepted

What feature engineering is necessary with tree based algorithms?

Feature engineering that I would consider essential for even tree based algorithms are: Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your ...
Eumenedies's user avatar
12 votes

When should I use StandardScaler and when MinMaxScaler?

In "Python Machine Learning" by Raschka the author provides some guidance on page 111 when to normalize (min-max scale) and when to standardize data: Although normalization via min-max ...
Jonathan's user avatar
  • 5,420
11 votes

Does scikit-learn have a forward selection/stepwise regression algorithm?

Sklearn DOES have a forward selection algorithm, although it isn't called that in scikit-learn. The feature selection method called F_regression in scikit-learn will sequentially include features ...
makansij's user avatar
  • 869
11 votes

is it possible to do feature selection for unsupervised machine learning problems?

Take a look at these links- https://stats.stackexchange.com/questions/108743/methods-in-r-or-python-to-perform-feature-selection-in-unsupervised-learning http://www.jmlr.org/papers/volume5/dy04a/...
Ankit Seth's user avatar
  • 1,821
11 votes

How to combine categorical and continuous input features for neural network training

There's three main approaches to solving this: Building two models separately and then training an ensemble algorithm that receives the output of the two models as an input Concating all the data ...
Tadej Magajna's user avatar
11 votes
Accepted

What does embedding mean in machine learning?

In the context of machine learning, an embedding is a low-dimensional, learned continuous vector representation of discrete variables into which you can translate high-dimensional vectors. Generally, ...
sdaylor's user avatar
  • 126
10 votes

List of feature engineering techniques

There is no definite source on how to do feature engineering. It is often dependent on the problem you are trying to solve. Some say it is more of an art than it is science. But I would go through ...
phiver's user avatar
  • 718
10 votes
Accepted

When to remove correlated variables

You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how ...
Björn's user avatar
  • 363
10 votes

Does feature selections matter to Decision Tree algorithms?

For ensembles of decision trees, feature selection is generally not that important. During the induction of decision trees, the optimal feature is selected to split the data based on metrics like ...
timleathart's user avatar
  • 3,940
10 votes
Accepted

Should features be correlated or uncorrelated for features-selection with the help of multiple regression analysis?

Q1) Should highly correlated features with the target variable be included or removed from classification and regression problems? Is there a better/elegant explanation to this step? Actually there's ...
Erwan's user avatar
  • 25.5k
9 votes

Any "rules of thumb" on number of features versus number of instances? (small data sets)

From my own experience:In one case, I worked with a real database that is very small (300 images) with many classes, severe data imbalance problem and I ended up with using 9 features: SIFT, HOG, ...
Bashar Haddad's user avatar
9 votes
Accepted

Difference between RFE and SelectFromModel in Scikit-Learn

They effectively try to achieve the same result but the methodology used by each technique varies a little. RFE removes least significant features over iterations. So basically it first removes a ...
user-116's user avatar
  • 671

Only top scored, non community-wiki answers of a minimum length are eligible