Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [feature-selection]

Methods and principles of selecting a subset of attributes for use in further modelling

Filter by
Sorted by
Tagged with
3 votes
1 answer
8k views

How to interpret Variance Inflation Factor (VIF) results?

From various books and blog posts, I understood that the Variance Inflation Factor (VIF) is used to calculate collinearity. They say that VIF till 10 is good. But I have a question. As we can see in ...
thewhitetulip's user avatar
59 votes
6 answers
61k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
neural-nut's user avatar
  • 1,783
70 votes
11 answers
40k views

What is dimensionality reduction? What is the difference between feature selection and extraction?

From wikipedia: dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature ...
alvas's user avatar
  • 2,410
63 votes
10 answers
67k views

Machine learning - features engineering from date/time data

What are the common/best practices to handle time data for machine learning application? For example, if in data set there is a column with timestamp of event, such as "2014-05-05", how you can ...
Igor Bobriakov's user avatar
60 votes
8 answers
128k views

Does scikit-learn have a forward selection/stepwise regression algorithm?

I am working on a problem with too many features and training my models takes way too long. I implemented a forward selection algorithm to choose features. However, I was wondering does scikit-learn ...
Maksud's user avatar
  • 725
10 votes
2 answers
7k views

Linear Regression and scaling of data

The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors). For mtcars dataset (here and ...
rnso's user avatar
  • 1,578
4 votes
1 answer
4k views

Cleaning input data with pd.get_dummies()

What is the advantage of converting a series like >>> df Color 0 Red 1 Blue 2 Green 3 Red To a multiple series like the below? ...
kaza's user avatar
  • 143
3 votes
1 answer
566 views

When should I oversample data?

I am dealing with multi-class classifiers. My data is unbalanced. Hence, I need to apply sampling techniques before training (undersampling or oversampling). When I apply undersampling, ...
Kyv's user avatar
  • 151
1 vote
0 answers
529 views

RFECV best n_features doesn't correspond to best gridscore

I am working on a feature selection for a binary classification problem with 977 records (and class proportion of 77:23). I already referred these two related posts - here and here. step size = 1 and ...
The Great's user avatar
  • 2,565
21 votes
5 answers
35k views

Feature selection vs Feature extraction. Which to use when?

Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am right. Which technique would be ...
Sid's user avatar
  • 677
18 votes
3 answers
44k views

How to determine feature importance in a neural network?

I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
Aesir's user avatar
  • 458
18 votes
3 answers
40k views

When should I use StandardScaler and when MinMaxScaler?

I have a feature vector with One-Hot-Encoded features and with continous features. How can I decide now, which data I shall scale with StandardScaler and which data scale with MinMaxScaler? I think I ...
jochen6677's user avatar
16 votes
4 answers
2k views

What are the implications for training a Tree Ensemble with highly biased datasets?

I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
gallamine's user avatar
  • 418
15 votes
3 answers
16k views

Is feature selection necessary?

I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a ...
LUSAQX's user avatar
  • 783
11 votes
3 answers
25k views

Can GPS coordinates (latitude and longitude) be used as features in a linear model?

I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and ...
stackoverflowuser2010's user avatar
11 votes
2 answers
15k views

Dissmissing features based on correlation with target variable

Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem? say for instance I have a dataset with the following format where the ...
MedAli's user avatar
  • 275
5 votes
2 answers
1k views

Decision Trees Should We Discard Low Importance Features?

I just started to work with feature selection. Let's say I have a decision tree model. I get its feature importances by tree.feature_importances_. In my model out ...
J.Smith's user avatar
  • 458
5 votes
3 answers
2k views

Categorizing Customer Emails

I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records ...
Pranay Mathur's user avatar
4 votes
2 answers
822 views

Repeated features in Neural Networks with tabular data

When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too ...
Carlos Mougan's user avatar
2 votes
2 answers
328 views

Feature Importance without Random Forest Feature Importances

Is their an intuitive way of finding feature importances without just using the random forest feature importances method? I have a binary logistic regression problem where I have binary features (1 or ...
Eisen's user avatar
  • 281
1 vote
2 answers
132 views

Regression Algorithms in Production

I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point. I have a few questions: Is feature selection the first step to take ...
Dwight K's user avatar
33 votes
6 answers
10k views

Are there any tools for feature engineering?

Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar ...
John's user avatar
  • 431
29 votes
3 answers
14k views

How to combine categorical and continuous input features for neural network training

Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
JunjieChen's user avatar
19 votes
1 answer
14k views

What is difference between one hot encoding and leave one out encoding?

I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences ...
icm's user avatar
  • 529
18 votes
2 answers
18k views

List of feature engineering techniques

Is there any resource with a list of feature engineering techniques? A mapping of type of data, model and feature engineering technique would be a gold mine.
icm's user avatar
  • 529
15 votes
2 answers
23k views

How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif

I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem. As I understand from the calculation of ANOVA from basic ...
JuniorDataGuyAKL's user avatar
14 votes
5 answers
26k views

When to remove correlated variables

Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?
bp89's user avatar
  • 143
13 votes
1 answer
17k views

Feature selection using feature importances in random forests with scikit-learn

I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to ...
Franck Dernoncourt's user avatar
12 votes
1 answer
5k views

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
user90772's user avatar
  • 331
12 votes
1 answer
10k views

Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67. I had the same algorithms and same set of hyperparamters during yesterday and ...
The Great's user avatar
  • 2,565
10 votes
3 answers
5k views

LightGBM - Why Exclusive Feature Bundling (EFB)?

I'm currently studying GBDT and started reading LightGBM's research paper. In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
Tom's user avatar
  • 103
7 votes
1 answer
7k views

How to interpret Shapley value plot for a model?

I was trying to use Shapley value approach for understanding the model predictions. I am trying this on a Xgboost model. My plot ...
The Great's user avatar
  • 2,565
7 votes
4 answers
27k views

is it possible to do feature selection for unsupervised machine learning problems?

I started looking for ways to do feature selection in machine learning. By having a quick look at this post , I made the assumption that feature selection is only manageable for supervised learning ...
ChiPlusPlus's user avatar
6 votes
3 answers
9k views

Should features be correlated or uncorrelated for features-selection with the help of multiple regression analysis?

I have seen researchers using Pearson correlation coefficient to find out the relevant features - to keep the features that have a high correlation value with the target. The implication is that the ...
Srishti M's user avatar
  • 471
6 votes
2 answers
3k views

RandomForest and tree feature importance in scikit-learn

What is the difference between model.feature_importances_ and tree.feature_importances_ in the following code: ...
dokondr's user avatar
  • 295
6 votes
1 answer
4k views

Difference between RFE and SelectFromModel in Scikit-Learn

What is the difference between Recursive Feature Elimination (RFE) function and SelectFromModel in Scikit-Learn? Both seems exactly similar.
raja's user avatar
  • 163
5 votes
2 answers
1k views

Elimination of features based on high covariance without affecting performance?

I ran into a question where the answer ran me into a big doubt. Suppose we have a dataset $A=${$x1,x2,y$} in which $x1$ and $x2$ are our features and $y$ is the label. Also, suppose that the ...
user307393's user avatar
5 votes
2 answers
5k views

Feature Selection with one-hot-encoded categorical data

I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem. My professors asked me to perform feature ...
dungeon's user avatar
  • 175
4 votes
3 answers
6k views

How to interpret shapley force plot for feature importance?

I am trying to practice and learn shapley value approach to explain my predictions on a binary classification problem. However am having difficulty in understanding the below plot. 1) Does it ...
The Great's user avatar
  • 2,565
4 votes
2 answers
139 views

Feature selection is not that useful?

I've been doing a few DataScience competitions now, and i'm noticing something quite odd and frustrating to me. Why is it frustrating? Because , in theory, when you read about datascience it's all ...
Blenz's user avatar
  • 2,084
4 votes
1 answer
3k views

Is there any difference between feature extraction and feature learning?

It appears to me that "feature extraction" and "feature learning" are equivalent concepts, however there are 2 separate wikipedia articles dedicated to them that are notably different. In particular, ...
Tnatsissa H Craeser's user avatar
3 votes
1 answer
849 views

How to visualize data of a multidimensional dataset (TIMIT)

I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 ...
giogix's user avatar
  • 163
3 votes
1 answer
245 views

Optimizing Expensive Functions

I'm trying some different techniques to optimise a Boosted Gradient Regressor by using an evolutionary programming technique to try and find the most efficient set of features. So far I've been having ...
Johnathan Brown's user avatar
3 votes
2 answers
7k views

How to implement feature selection for categorical variables (especially with many categories)?

I've been trying to get some ideas of how I could treat categorical variables when doing feature selection. Mainly I've been running Random Forest feature importance on Python for which preprocessing ...
rayven1lk's user avatar
  • 371
3 votes
1 answer
9k views

What measures can I use to find correlation between categorical features and binary label?

For analyzing numerical features, we have correlation. What measures do we have to analyse the relevance of a categorical feature to the target value? If there isn't a direct measure, how can we ...
ProgramSpree's user avatar
2 votes
1 answer
198 views

Feature selection where adding features are deteriorating model

k I am training a kNN classifier with 144 features and graphed the accuracy vs number of features used and got this. What might be the reason for the drops in the accuracy at some points of the graph?...
mc8's user avatar
  • 145
2 votes
2 answers
344 views

How Decision Tree Classifier works? [closed]

In particular i am using SKLearn with class DecisionTreeClassifier. I would really like to understand how the tree build itself ...
baltiturg's user avatar
  • 143
2 votes
2 answers
1k views

How to select features when performing classification with a dataframe of multiple columns?

I have a dataframe of 50000 observations and I want to perform a classification task. But I'm struggling with features selection. I have 89 columns, which after getting rid of some redundant features, ...
Dimi's user avatar
  • 223
2 votes
2 answers
84 views

Different approaches to label data

I have a dataset of patient records but they don't have labels. I would like to label them and would like to know what are the different approaches available that I can consider to label them. For ...
The Great's user avatar
  • 2,565
2 votes
1 answer
351 views

How to build a model when we have three separate train, validation, and test sets?

I have a data set which should be divided into train, test, and validation sets. ...
ebrahimi's user avatar
  • 1,307