55
Simply put:
feature selection: you select a subset of the original feature set; while
feature extraction: you build a new set of features from the original feature set.
Examples of feature extraction: extraction of contours in images, extraction of digrams from a text, extraction of phonemes from recording of spoken text, etc.
Feature extraction ...
40
The most logical way to transform hour is into two variables that swing back and forth out of sink. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and forth out of sink with the y position. For a 24-hour clock you can accomplish this with x=sin(2pi*hour/24),y=cos(2pi*hour/24).
You need both variables or the ...
29
Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through.
(More Info)
23
Very interesting question (+1). While I am not aware of any software tools that currently offer comprehensive functionality for feature engineering, there is definitely a wide range of options in that regard. Currently, as far as I know, feature engineering is still largely a laborious and manual process (i.e., see this blog post). Speaking about the feature ...
23
Adding to The answer given by Toros,
These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember)
feature extraction and feature engineering: transformation of raw data into features suitable for modeling;
feature transformation: transformation of data to improve the accuracy of the algorithm;
feature ...
21
We love the normal form
In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view!
Which Transformation?
The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions.
What makes physical (biological, ...
19
You do not need domain knowledge (the knowledge of what your data mean) in order to do feature engineering (finding more expressive ways of framing your data).
As Tu N. explained, you can find "quick and dirty" combinations of features that could be helpful pretty easily. Given an output $y$ and an individual feature $x$, you can take the following ...
19
You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$".
For your coefficients to be interpretable, linear regression assumes a bunch of things.
One of these things is no multicollinearity. That is, your $X$ ...
18
Dimensionality reduction is typically choosing a basis or mathematical representation within which you can describe most but not all of the variance within your data, thereby retaining the relevant information, while reducing the amount of information necessary to represent it. There are a variety of techniques for doing this including but not limited to PCA,...
16
A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information. A low correlation between the new feature and existing features is likely preferable.
A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be ...
16
They are probably using "leave one out encoding" to refer to Owen Zhang's strategy.
From here
The encoded column is not a conventional dummy variable, but instead is the mean response over all rows for this categorical level, excluding the row itself. This gives you the advantage of having a one-column representation of the categorical while ...
15
The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable.
If you encode time as numeric, then you are imposing certain restrictions on the model. For a linear regression model, the effect of time is now monotonic, either the target will increase or decrease with time. For decision trees, ...
14
In the general case, this is by no means true. Let's break down the case for different data scenarios:
For discriminative image models (e.g. image classification/labeling) this is true for some scenarios. You just throw some convnets (even pretrained models) at your data, and that's it. Nevertheless, convnets themselves profit from the "expert knowledge" ...
12
In images, some frequently used techniques for feature extraction are binarizing and blurring
Binarizing: converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.
This is how you do ...
11
No, manual feature extraction is not outdated.
In addition, manual feature extraction is hard to do-away, given, a data scientist needs business and domain logic to build a robust model to replicate and capture trend and pattern from data. Nevertheless, there are exceptions such as image data.
Depends, if its image data, yes the statement is true. There ...
10
There is no definite source on how to do feature engineering. It is often dependent on the problem you are trying to solve. Some say it is more of an art than it is science.
But I would go through some of the high scoring kaggle kernels / winning solutions if available. Just head over to kaggle and browse through the competitions. There is a lot of very ...
9
By using a parse tree, you divide your sentence into parts. Suppose, in the example of sentiment analysis, you can use those parts to assign a positive/negative sentiment to each part and then take the cumulative effect of those parts.
This image will help you understand more. The first half has a negative sentiment(mainly because of the word "dry") but ...
9
First off, ignore the haters. I started working on ML in Music a long time ago and got several degrees using that work. When I started I was asking people the same kind of questions you are. It is a fascinating field and there is always room for someone new. We all have to start somewhere.
The areas of study you are inquiring about are Music Information ...
8
As you've described it, Step 4 is where you want to use TF-IDF. Essentially, TD-IDF will count each term in each document, and assign a score given the relative frequency across the collection of documents.
There's one big step missing from your process, however: annotating a training set. Before you train your classifier, you'll need to manually annotate a ...
8
This great tutorial covers the basics of convolutional neuraltworks, which are currently achieving state of the art performance in most vision tasks:
http://deeplearning.net/tutorial/lenet.html
There are a number of options for CNNs in python, including Theano and the libraries built on top of it (I found keras to be easy to use).
If you prefer to avoid ...
8
I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it is to category 3). This is definitely not the case for hours or months.
However, the issue that you raise is that you want to represent hours and months in a ...
8
From the documentation:
Similar to AgglomerativeClustering, but recursively merges features instead of samples.
In standard agglomerative clustering you receive a matrix $M^{n \times m}$ representing $n$ samples of dimension $m$ that you want to cluster. In feature agglomeration the algorithm clusters the transpose of the matrix, i.e. $M^T$ so it ...
8
This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply:
data_df[data_df > 15000]
If this data is sitting in a particular column, you can use this instead:
data_df[data_df['column_name'] > 15000]
These will return the peak values.
Updated Answer:
If ...
7
As in @damienfrancois answer feature selection is about selecting a subset of features. So in NLP it would be selecting a set of specific words (the typical in NLP is that each word represents a feature with value equal to the frequency of the word or some other weight based on TF/IDF or similar).
Dimensionality reduction is the introduction of new feature ...
7
Featuretools is a recently released python library for automated feature engineering. It's based on an algorithm called Deep Feature Synthesis originally developed in 2015 MIT and tested on public data science competitions on Kaggle.
Here is how it fits into the common data science process.
The aim of the library is to not only help experts build better ...
7
This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use frequency or the cdf of the frequency (to avoid noise generated by heavy-tailed pdf), and they achieved pretty good results with lightGBM. However, i cannot really ...
7
The issue with building a regression model on all 3 of these is that you are potentially introducing multicollinearity into the model. Although log(input) and sqrt(input) are not linear functions of the input a quick test (using Matlab) shows they are still highly correlated (depending on the range)
input=rand(1,100);
input_log=log(input);
input_rt=sqrt(...
6
Yes, it is entirely possible to combine unsupervised learning with the CRF model. In particular, I would recommend that you explore the possibility of using word2vec features as inputs to your CRF.
Word2vec trains a to distinguish between words that are appropriate for a given context and words that are randomly selected. Select weights of the model can ...
6
One-hot-encoded ZIP codes shouldn't present a problem with modern tools, where features can be much wider (millions, billions even), but if you really want you could aggregate area codes into regions, such as states. Of course, you should not use strings, but bit vectors. Two other dimensionality reduction options are MCA (PCA for categorical variables) and ...
6
Boruta is by universal reputation dog-slow and not very good. Boruta runs take many hours or days. VIF feature-selection algorithm is not objective, anyway.
You can program your own feature-selection that runs faster. I ran Boruta a few times on various datasets and it wasted 4 days of my time, and the result was inconclusive.
Here's the fast-and-dirty not-...
Only top voted, non community-wiki answers of a minimum length are eligible
