Questions tagged [feature-selection]

Methods and principles of selecting a subset of attributes for use in further modelling

Filter by
Sorted by
Tagged with
60
votes
11answers
32k views

What is dimensionality reduction? What is the difference between feature selection and extraction?

From wikipedia, dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and ...
46
votes
10answers
39k views

Machine learning - features engineering from date/time data

What are the common/best practices to handle time data for machine learning application? For example, if in data set there is a column with timestamp of event, such as "2014-05-05", how you can ...
38
votes
5answers
61k views

Does scikit-learn have forward selection/stepwise regression algorithm?

I'm working on the problem with too many features and training my models takes way too long. I implemented forward selection algorithm to choose features. However, I was wondering does scikit-learn ...
29
votes
6answers
8k views

Are there any tools for feature engineering?

Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar ...
23
votes
4answers
15k views

Does XGBoost handle multicollinearity by itself?

I'm currently using XGBoost on a data-set with 21 features (selected from list of some 150 features), then one-hot coded them to obtain ~98 features. A few of these 98 features are somewhat redundant, ...
19
votes
3answers
11k views

How to perform feature engineering on unknown features?

I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers. People are performing ...
19
votes
2answers
11k views

Text categorization: combining different kind of features

The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). ...
16
votes
4answers
13k views

Any “rules of thumb” on number of features versus number of instances? (small data sets)

I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By ...
16
votes
2answers
5k views

How to choose the features for a neural network?

I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to ...
15
votes
4answers
4k views

How to specify important attributes?

Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to ...
14
votes
4answers
2k views

What are the implications for training a Tree Ensemble with highly biased datasets?

I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...
14
votes
2answers
3k views

How to combine categorical and continuous input features for neural network training

Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. ...
13
votes
2answers
4k views

What features are generally used from Parse trees in classification process in NLP?

I am exploring different types of parse tree structures. The two widely known parse tree structures are a) Constituency based parse tree and b) Dependency based parse tree structures. I am able to ...
12
votes
5answers
18k views

Feature selection vs Feature extraction. Which to use when?

Feature extraction and feature selection essentially reduce the dimensionality of the data, but feature extraction also makes the data more separable, if I am right. Which technique would be ...
12
votes
1answer
6k views

What is difference between one hot encoding and leave one out encoding?

I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences ...
12
votes
1answer
3k views

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

I was trying to use feature importances from Random Forests to perform some empirical feature selection for a regression problem where all the features are categorical and a lot of them have many ...
11
votes
1answer
15k views

Feature selection using feature importances in random forests with scikit-learn

I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to ...
11
votes
4answers
3k views

Feature Extraction Technique - Summarizing a Sequence of Data

I often am building a model (classification or regression) where I have some predictor variables that are sequences and I have been trying to find technique recommendations for summarizing them in the ...
11
votes
4answers
3k views

Which one first: algorithm benchmarking, feature selection, parameter tuning?

When trying to do e.g. a classification, my approach currently is to try out various algorithm first and benchmark them perform feature selection on the best algorithm from 1 above tune the ...
11
votes
2answers
758 views

Linear Regression and scaling of data

The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors). For mtcars dataset (here and ...
10
votes
4answers
9k views

Feature selection and classification accuracy relation

One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your ...
10
votes
7answers
2k views

Data science projects explained step by step?

I am looking for a website or book where several practical examples are given step by step, explaining how they choose the relevant features, the model selection procedure, etc...
10
votes
4answers
1k views

How to compare the performance of feature selection methods?

There are several feature selection / variable selection approaches (see for example Guyon & Elisseeff, 2003; Liu et al., 2010): filter methods (e.g., correlation-based, entropy-based, random ...
9
votes
3answers
9k views

Is feature selection necessary?

I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a ...
9
votes
5answers
4k views

When to remove correlated variables

Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?
9
votes
1answer
26k views

How to do stepwise regression using sklearn? [duplicate]

I could not find a way to stepwise regression in scikit learn. I have checked all other posts on Stack Exchange on this topic. Answers to all of them suggests using f_regression. But f_regression ...
9
votes
3answers
12k views

Can GPS coordinates (latitude and longitude) be used as features in a linear model?

I have data sets that contain, among many features, GPS coordinates (latitude and longitude). I'd like to use these data sets to explore problems such as: (1) computing ETA to drive between start and ...
9
votes
2answers
2k views

What to do when testing data has less features than training data?

Let's say we are predicting the sales of a shop and my training data has two sets of features: One about the store sales with the dates (the field "Store" is not unique) One about the store types (...
9
votes
1answer
3k views

feature importance via random forest and linear regression are different

Applied Lasso to rank the features and got the following results: ...
9
votes
1answer
760 views

Feature selection for Support Vector Machines

My question is three-fold In the context of "Kernelized" support vector machines Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and ...
9
votes
1answer
176 views

Learning signal encoding

I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there ...
8
votes
3answers
135 views

Feature selection for tracking user activity within an application

I am developing a system that is intended to capture the "context" of user activity within an application; it is a framework that web applications can use to tag user activity based on requests made ...
8
votes
2answers
11k views

List of feature engineering techniques

Is there any resource with a list of feature engineering techniques? A mapping of type of data, model and feature engineering technique would be a gold mine
8
votes
1answer
1k views

Document classification: tf-idf prior to or after feature filtering?

I have a document classification project where I am getting site content and then assigning one of numerous labels to the website according to content. I found out that tf-idf could be very useful ...
8
votes
4answers
120 views

How to handle features which are not always available?

I have a feature in my feature vector that is not always available respectively sometimes (for some samples) it makes no sense to use it. I feed a sklearn MLPClassifier with this feature vector. Does ...
6
votes
2answers
990 views

Improve a regression model and feature selection

I am working on Azure ML Studio and try to create a regression model to predict a numerical value. I will try to describe my features and what I have done until now. My data with about 3 million rows ...
6
votes
2answers
11k views

Improving accuracy of Text Classification

I am working on a text classification problem, the objective is to classify news articles to their corresponding categories, but in this case the categories are not very broad like, politics, sports, ...
6
votes
3answers
694 views

Regression vs Random Forest - Combination of features

I had a discussion with a friend and we were talking about the advantages of random forest over linear regression. At some point, my friend said that one of the advantages of the random forest over ...
6
votes
1answer
780 views

Interpreting the results of randomized PCA in scikit-learn

I'm using scikit-learn to do a genome-wide association study with a feature vector of about 100K SNPs. My goal is to tell the biologists which SNPs are "interesting". RandomizedPCA really improved ...
6
votes
1answer
95 views

Will unnecessary features harm the tree based model?

Is it necessary to drop noisy features (eg column of random numbers) from tree features? I think it's not. sometimes it may benefit but will never cause any harm to the model. Because at each split ...
6
votes
1answer
1k views

Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution?

I have a categorical variable which has thousands of values, for a dataset which has millions of records. The data is being used to create a binary classification model. I am in the early steps of ...
6
votes
2answers
432 views

Image segmentation - handcrafted features vs DNN?

Currently working on a project that requires multi-class image segmentation to identify distinct anomaly types in various sheet metals. Have had moderate success with various NN segmentation ...
6
votes
1answer
2k views

Named entity recognition (NER) features

I'm new to Named Entity Recognition and I'm having some trouble understanding what/how features are used for this task. Some papers I've read so far mention features used, but don't really explain ...
6
votes
1answer
2k views

How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?

I'm building a recommender system and using SVD as one of the preprocessing techniques. However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (...
6
votes
1answer
2k views

LSTM Feature selection process

We need to implement a time series problem by LSTM model. But, while implementing the same, the main challenge I am facing is feature selection issue. Because, our data-set contain 2300 observations ...
5
votes
4answers
328 views

General way to reduce features

Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run. What are some of the best ways to find the most influential columns/...
5
votes
3answers
5k views

feature selection techniques

Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?
5
votes
2answers
8k views

Dissmissing features based on correlation with target variable

Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem? say for instance I have a dataset with the following format where the ...
5
votes
3answers
698 views

Is automatic feature detection feasible?

I am searching for pointers to algorithms for feature detection. EDIT: all the answers helped me a lot, I cannot decide which one I should accept. THX guys! What I did: For discrete variables (i.e....
5
votes
1answer
112 views

Why is duplicating inputs bad?

I am trying to predict an output value based on several continuously-valued inputs using a regression model. I am not sure what approach is appropriate to scale/transform the input data for the ...