Questions tagged [feature-selection]
Methods and principles of selecting a subset of attributes for use in further modelling
964
questions
70
votes
11
answers
40k
views
What is dimensionality reduction? What is the difference between feature selection and extraction?
From wikipedia:
dimensionality reduction or dimension reduction is the process of
reducing the number of random variables under consideration, and
can be divided into feature selection and feature ...
- 2,340
63
votes
10
answers
67k
views
Machine learning - features engineering from date/time data
What are the common/best practices to handle time data for machine learning application?
For example, if in data set there is a column with timestamp of event, such as "2014-05-05", how you can ...
- 1,071
60
votes
8
answers
125k
views
Does scikit-learn have a forward selection/stepwise regression algorithm?
I am working on a problem with too many features and training my models takes way too long. I implemented a forward selection algorithm to choose features.
However, I was wondering does scikit-learn ...
- 725
59
votes
6
answers
57k
views
Does XGBoost handle multicollinearity by itself?
I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
- 1,783
33
votes
6
answers
10k
views
Are there any tools for feature engineering?
Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar ...
- 431
28
votes
3
answers
13k
views
How to combine categorical and continuous input features for neural network training
Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
- 515
26
votes
4
answers
47k
views
Any "rules of thumb" on number of features versus number of instances? (small data sets)
I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By ...
- 483
26
votes
2
answers
18k
views
Text categorization: combining different kind of features
The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). ...
- 361
22
votes
3
answers
13k
views
How to perform feature engineering on unknown features?
I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.
People are performing ...
- 451
22
votes
2
answers
7k
views
How to choose the features for a neural network?
I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way ...
- 1,667
21
votes
5
answers
35k
views
Feature selection vs Feature extraction. Which to use when?
Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am right.
Which technique would be ...
- 667
21
votes
6
answers
31k
views
What does embedding mean in machine learning?
I just met a terminology called "embedding" in a paper regarding deep learning. The context is "multi-modal embedding"
My guess: embedding of something is extract some feature of sth,to form a vector....
- 393
19
votes
1
answer
14k
views
What is difference between one hot encoding and leave one out encoding?
I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences ...
- 529
18
votes
3
answers
39k
views
How to determine feature importance in a neural network?
I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
- 448
18
votes
2
answers
18k
views
List of feature engineering techniques
Is there any resource with a list of feature engineering techniques? A mapping of type of data, model and feature engineering technique would be a gold mine.
- 529
18
votes
3
answers
37k
views
When should I use StandardScaler and when MinMaxScaler?
I have a feature vector with One-Hot-Encoded features and with continous features.
How can I decide now, which data I shall scale with StandardScaler and which data scale with MinMaxScaler? I think I ...
- 571
16
votes
4
answers
2k
views
What are the implications for training a Tree Ensemble with highly biased datasets?
I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
- 418
15
votes
2
answers
21k
views
How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif
I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem.
As I understand from the calculation of ANOVA from basic ...
- 153
15
votes
4
answers
6k
views
How to specify important attributes?
Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to ...
- 503
15
votes
3
answers
16k
views
Is feature selection necessary?
I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a ...
- 783
15
votes
4
answers
3k
views
How to compare the performance of feature selection methods?
There are several feature selection / variable selection approaches (see for example Guyon & Elisseeff, 2003; Liu et al., 2010):
filter methods (e.g., correlation-based, entropy-based, random ...
- 341
14
votes
5
answers
23k
views
When to remove correlated variables
Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?
- 143
14
votes
2
answers
7k
views
What to do when testing data has less features than training data?
Let's say we are predicting the sales of a shop and my training data has two sets of features:
One about the store sales with the dates (the field "Store" is not unique)
One about the store types (...
- 2,340
13
votes
1
answer
57k
views
How to do stepwise regression using sklearn? [duplicate]
I could not find a way to stepwise regression in scikit learn. I have checked all other posts on Stack Exchange on this topic. Answers to all of them suggests using f_regression.
But f_regression ...
- 131
13
votes
1
answer
17k
views
Feature selection using feature importances in random forests with scikit-learn
I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to ...
- 5,630
13
votes
2
answers
4k
views
What features are generally used from Parse trees in classification process in NLP?
I am exploring different types of parse tree structures. The two widely known parse tree structures are
a) Constituency based parse tree and
b) Dependency based parse tree structures.
I am able to ...
- 231
12
votes
1
answer
9k
views
Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn
I have been running few ML models on same set of data for a binary classification problem with class proportion of 33:67.
I had the same algorithms and same set of hyperparamters during yesterday and ...
- 2,555
12
votes
1
answer
7k
views
What feature engineering is necessary with tree based algorithms?
I understand data hygiene, which is probably the most basic feature engineering. That is making sure all your data is properly loaded, making sure N/As are treated ...
- 423
12
votes
1
answer
4k
views
Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)
I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
- 331
11
votes
2
answers
15k
views
Dissmissing features based on correlation with target variable
Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem?
say for instance I have a dataset with the following format where the ...
- 275
11
votes
4
answers
4k
views
Feature Extraction Technique - Summarizing a Sequence of Data
I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the ...
- 702
11
votes
4
answers
5k
views
Which one first: algorithm benchmarking, feature selection, parameter tuning?
When trying to do e.g. a classification, my approach currently is to
try out various algorithm first and benchmark them
perform feature selection on the best algorithm from 1 above
tune the ...
- 269
11
votes
4
answers
10k
views
Feature selection and classification accuracy relation
One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your ...
- 113
11
votes
3
answers
24k
views
Can GPS coordinates (latitude and longitude) be used as features in a linear model?
I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and ...
10
votes
2
answers
6k
views
Linear Regression and scaling of data
The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors).
For mtcars dataset (here and ...
- 1,558
10
votes
3
answers
5k
views
LightGBM - Why Exclusive Feature Bundling (EFB)?
I'm currently studying GBDT and started reading LightGBM's research paper.
In section 4. they explain the Exclusive Feature Bundling algorithm, which aims at reducing the number of features by ...
- 103
10
votes
2
answers
16k
views
How to get feature importance from a keras deep learning model?
In case of scikit-learn's models, we can get feature importance using the relevant attributes of the model.
I've been working on a RNN, using LSTMs for text embedding.
Is there any way to get ...
- 140
10
votes
7
answers
3k
views
Data science projects explained step by step?
I am looking for a website or book where several practical examples are given step by step, explaining how they choose the relevant features, the model selection procedure, etc...
- 807
9
votes
2
answers
2k
views
Why continuous features are more important than categorical features in decision tree models?
I have both categorical and continuous features in my prediction model and want to select (and rank) most important features.
I have converted all categorical variables into dummy variables using one ...
- 103
9
votes
2
answers
13k
views
Does feature selections matter to Decision Tree algorithms?
Background: Currently I'm working on my thesis project, which is to build Tree-based ensemble methods for classification on a large data set. Before I started with modeling, I've spent a large amount ...
- 91
9
votes
2
answers
7k
views
What is the rationale for discretization of continuous features and when should it be done?
Continous feature discretization usually leads to lose of information due to the binning process. However most of the Top solutions for Kaggle Titanic are based on discretization(age,fare).
When ...
- 91
9
votes
4
answers
23k
views
Improving accuracy of Text Classification
I am working on a text classification problem, the objective is to classify news articles to their corresponding categories, but in this case the categories are not very broad like, politics, sports, ...
- 159
9
votes
2
answers
2k
views
Does "feature importance" depend on the model type?
I was working on a small classification problem (breast cancer data set from sklearn), and trying to decide which features were most important to predict the labels. I understand that there are ...
- 200
9
votes
1
answer
5k
views
feature importance via random forest and linear regression are different
Applied Lasso to rank the features and got the following results:
...
- 193
9
votes
2
answers
24k
views
Always drop the first column after performing One Hot Encoding?
Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
- 403
9
votes
2
answers
4k
views
LSTM Feature selection process
We need to implement a time series problem with the LSTM model.
But, while implementing the same, the main challenge I am facing is the feature selection issue. Because our data-set contains 2300 ...
9
votes
1
answer
1k
views
Feature selection for Support Vector Machines
My question is three-fold
In the context of "Kernelized" support vector machines
Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and ...
9
votes
1
answer
257
views
Learning signal encoding
I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
- 1,824
9
votes
2
answers
206
views
Features reduction for the not correlated data set
I am working with classification problem on a training data set, which have 100 features. All the features in pairs haven't visible correlation. One can see it in the example pair plot for the some of ...
- 211
8
votes
3
answers
166
views
Feature selection for tracking user activity within an application
I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made ...
- 181