Questions tagged [feature-selection]
Methods and principles of selecting a subset of attributes for use in further modelling
65
questions
3
votes
1
answer
8k
views
How to interpret Variance Inflation Factor (VIF) results?
From various books and blog posts, I understood that the Variance Inflation Factor (VIF) is used to calculate collinearity. They say that VIF till 10 is good. But I have a question.
As we can see in ...
- 153
59
votes
6
answers
64k
views
Does XGBoost handle multicollinearity by itself?
I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
- 1,783
70
votes
11
answers
40k
views
What is dimensionality reduction? What is the difference between feature selection and extraction?
From wikipedia:
dimensionality reduction or dimension reduction is the process of
reducing the number of random variables under consideration, and
can be divided into feature selection and feature ...
- 2,430
63
votes
10
answers
68k
views
Machine learning - features engineering from date/time data
What are the common/best practices to handle time data for machine learning application?
For example, if in data set there is a column with timestamp of event, such as "2014-05-05", how you can ...
- 1,071
60
votes
8
answers
129k
views
Does scikit-learn have a forward selection/stepwise regression algorithm?
I am working on a problem with too many features and training my models takes way too long. I implemented a forward selection algorithm to choose features.
However, I was wondering does scikit-learn ...
- 725
18
votes
3
answers
42k
views
When should I use StandardScaler and when MinMaxScaler?
I have a feature vector with One-Hot-Encoded features and with continous features.
How can I decide now, which data I shall scale with StandardScaler and which data scale with MinMaxScaler? I think I ...
- 601
10
votes
2
answers
7k
views
Linear Regression and scaling of data
The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors).
For mtcars dataset (here and ...
- 1,578
4
votes
1
answer
4k
views
Cleaning input data with pd.get_dummies()
What is the advantage of converting a series like
>>> df
Color
0 Red
1 Blue
2 Green
3 Red
To a multiple series like the below?
...
- 143
3
votes
1
answer
615
views
When should I oversample data?
I am dealing with multi-class classifiers. My data is unbalanced. Hence, I need to apply sampling techniques before training (undersampling or oversampling). When I apply undersampling, ...
- 151
1
vote
0
answers
581
views
RFECV best n_features doesn't correspond to best gridscore
I am working on a feature selection for a binary classification problem with 977 records (and class proportion of 77:23). I already referred these two related posts - here and here. step size = 1 and ...
- 2,607
21
votes
5
answers
35k
views
Feature selection vs Feature extraction. Which to use when?
Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am right.
Which technique would be ...
- 677
18
votes
3
answers
47k
views
How to determine feature importance in a neural network?
I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
- 458
16
votes
4
answers
2k
views
What are the implications for training a Tree Ensemble with highly biased datasets?
I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
- 418
15
votes
3
answers
17k
views
Is feature selection necessary?
I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a ...
- 783
11
votes
2
answers
15k
views
Dissmissing features based on correlation with target variable
Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem?
say for instance I have a dataset with the following format where the ...
- 275
11
votes
3
answers
25k
views
Can GPS coordinates (latitude and longitude) be used as features in a linear model?
I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and ...
5
votes
2
answers
1k
views
Decision Trees Should We Discard Low Importance Features?
I just started to work with feature selection. Let's say I have a decision tree model. I get its feature importances by tree.feature_importances_.
In my model out ...
- 468
5
votes
3
answers
2k
views
Categorizing Customer Emails
I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records ...
- 185
4
votes
2
answers
883
views
Repeated features in Neural Networks with tabular data
When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too ...
- 6,332
2
votes
2
answers
398
views
Feature Importance without Random Forest Feature Importances
Is their an intuitive way of finding feature importances without just using the random forest feature importances method?
I have a binary logistic regression problem where I have binary features (1 or ...
- 281
1
vote
2
answers
132
views
Regression Algorithms in Production
I am interested in predicting if a doctor would prescribe a specific drug and have chosen Logistic Regression as a starting point.
I have a few questions:
Is feature selection the first step to take ...
- 11
33
votes
6
answers
10k
views
Are there any tools for feature engineering?
Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar ...
- 431
29
votes
3
answers
14k
views
How to combine categorical and continuous input features for neural network training
Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
- 525
20
votes
1
answer
14k
views
What is difference between one hot encoding and leave one out encoding?
I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences ...
- 539
18
votes
2
answers
18k
views
List of feature engineering techniques
Is there any resource with a list of feature engineering techniques? A mapping of type of data, model and feature engineering technique would be a gold mine.
- 539
16
votes
2
answers
24k
views
How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif
I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem.
As I understand from the calculation of ANOVA from basic ...
- 163
15
votes
5
answers
27k
views
When to remove correlated variables
Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?
- 153
13
votes
1
answer
10k
views
Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn
I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67.
I had the same algorithms and same set of hyperparamters during yesterday and ...
- 2,607
13
votes
1
answer
17k
views
Feature selection using feature importances in random forests with scikit-learn
I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to ...
- 5,750
12
votes
1
answer
5k
views
Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)
I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
- 331
10
votes
3
answers
6k
views
LightGBM - Why Exclusive Feature Bundling (EFB)?
I'm currently studying GBDT and started reading LightGBM's research paper.
In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
- 103
7
votes
4
answers
27k
views
is it possible to do feature selection for unsupervised machine learning problems?
I started looking for ways to do feature selection in machine learning.
By having a quick look at this post , I made the assumption that feature selection is only manageable for supervised learning ...
- 547
7
votes
1
answer
8k
views
How to interpret Shapley value plot for a model?
I was trying to use Shapley value approach for understanding the model predictions. I am trying this on a Xgboost model. My plot ...
- 2,607
6
votes
1
answer
4k
views
Difference between RFE and SelectFromModel in Scikit-Learn
What is the difference between Recursive Feature Elimination (RFE) function and SelectFromModel in Scikit-Learn? Both seems exactly similar.
- 163
6
votes
2
answers
3k
views
RandomForest and tree feature importance in scikit-learn
What is the difference between model.feature_importances_ and tree.feature_importances_ in the following code:
...
- 295
6
votes
3
answers
9k
views
Should features be correlated or uncorrelated for features-selection with the help of multiple regression analysis?
I have seen researchers using Pearson correlation coefficient to find out the relevant features - to keep the features that have a high correlation value with the target. The implication is that the ...
- 471
5
votes
2
answers
5k
views
Feature Selection with one-hot-encoded categorical data
I have a dataset with 400+ columns. Almost 90% of these are categorical data with One-Hot-Encoding (OHE). I'm using the dataset for a classification problem.
My professors asked me to perform feature ...
- 175
5
votes
2
answers
1k
views
Elimination of features based on high covariance without affecting performance?
I ran into a question where the answer ran me into a big doubt.
Suppose we have a dataset $A=${$x1,x2,y$} in which $x1$ and $x2$ are our features and $y$ is the label.
Also, suppose that the ...
- 51
4
votes
3
answers
6k
views
How to interpret shapley force plot for feature importance?
I am trying to practice and learn shapley value approach to explain my predictions on a binary classification problem. However am having difficulty in understanding the below plot.
1) Does it ...
- 2,607
4
votes
1
answer
3k
views
Is there any difference between feature extraction and feature learning?
It appears to me that "feature extraction" and "feature learning" are equivalent concepts, however there are 2 separate wikipedia articles dedicated to them that are notably different. In particular, ...
4
votes
2
answers
167
views
Feature selection is not that useful?
I've been doing a few DataScience competitions now, and i'm noticing something quite odd and frustrating to me. Why is it frustrating? Because , in theory, when you read about datascience it's all ...
- 2,094
3
votes
1
answer
856
views
How to visualize data of a multidimensional dataset (TIMIT)
I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 ...
- 163
3
votes
1
answer
254
views
Optimizing Expensive Functions
I'm trying some different techniques to optimise a Boosted Gradient Regressor by using an evolutionary programming technique to try and find the most efficient set of features. So far I've been having ...
- 131
3
votes
2
answers
7k
views
How to implement feature selection for categorical variables (especially with many categories)?
I've been trying to get some ideas of how I could treat categorical variables when doing feature selection. Mainly I've been running Random Forest feature importance on Python for which preprocessing ...
- 371
3
votes
1
answer
9k
views
What measures can I use to find correlation between categorical features and binary label?
For analyzing numerical features, we have correlation. What measures do we have to analyse the relevance of a categorical feature to the target value? If there isn't a direct measure, how can we ...
- 143
2
votes
1
answer
151
views
Ongoing feature selection
If you have a set of n features you have 2^n-1 non-empty feature subsets. As a result, if you pick one of them you are unlikely to have found the best one.
To me, it seems intuitive that as you build ...
- 181
2
votes
2
answers
354
views
How Decision Tree Classifier works? [closed]
In particular i am using SKLearn with class DecisionTreeClassifier.
I would really like to understand how the tree build itself ...
- 143
2
votes
1
answer
358
views
How to build a model when we have three separate train, validation, and test sets?
I have a data set which should be divided into train, test, and validation sets.
...
- 1,307
2
votes
2
answers
88
views
Different approaches to label data
I have a dataset of patient records but they don't have labels.
I would like to label them and would like to know what are the different approaches available that I can consider to label them.
For ...
- 2,607
2
votes
3
answers
682
views
Feature Selection and PCA
I have a classification problem. I want to reduce number of features to 4 (I have 30). I'm wondering why I get better result in classification when I use correlation based feature selection(cfs) first ...
- 453