183

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality. LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with ...


44

While AN6U5 has given a very good answer, I wanted to add a few points for future reference. When considering One Hot Encoding(OHE) and Label Encoding, we must try and understand what model you are trying to build. Namely the two categories of model we will be considering are: Tree Based Models: Gradient Boosted Decision Trees and Random Forests. Non-Tree ...


30

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through. (More Info)


27

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points ...


26

I was learning this topic too, and these are what I found: This type of encoding is called likelihood encoding, impact coding or target coding The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable ...


19

You do not need domain knowledge (the knowledge of what your data mean) in order to do feature engineering (finding more expressive ways of framing your data). As Tu N. explained, you can find "quick and dirty" combinations of features that could be helpful pretty easily. Given an output $y$ and an individual feature $x$, you can take the following ...


19

You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$". For your coefficients to be interpretable, linear regression assumes a bunch of things. One of these things is no multicollinearity. That is, your $X$ ...


17

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns. The effect of doing so will depend on the model class, and type of normalisation you apply, but I ...


16

They are probably using "leave one out encoding" to refer to Owen Zhang's strategy. From here The encoded column is not a conventional dummy variable, but instead is the mean response over all rows for this categorical level, excluding the row itself. This gives you the advantage of having a one-column representation of the categorical while ...


16

The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable. If you encode time as numeric, then you are imposing certain restrictions on the model. For a linear regression model, the effect of time is now monotonic, either the target will increase or decrease with time. For decision trees, ...


14

Let's define first Feature Engineering: Feature selection Feature extraction Adding features through domain expertise XGBoost does (1) for you. XGBoost does not do (2)/(3) for you. So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.


13

You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start. Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts ...


13

In my experience, when people claim to have an automated approach to feature engineering, they really mean "feature generation", and what they're actually talking about is that they've built a deep neural network of some sort. To be fair, in a limited sense, this could be a true claim. Properly trained deep neural networks can handle any number of pairwise ...


11

Feature selection: XGBoost does the feature selection up to a level. In my experience, I always do feature selection by a round of xgboost with parameters different than what I use for the final model. I typically use low numbers for row and feature sampling, and trees that are not deep and only keep the features that enter to the model. Then fine tune with ...


10

There is no definite source on how to do feature engineering. It is often dependent on the problem you are trying to solve. Some say it is more of an art than it is science. But I would go through some of the high scoring kaggle kernels / winning solutions if available. Just head over to kaggle and browse through the competitions. There is a lot of very ...


8

Target encoding is now available in sklearn through the category_encoders package. Target Encoder class category_encoders.target_encoder.TargetEncoder(verbose=0, cols=None, drop_invariant=False, return_df=True, impute_missing=True, handle_unknown='impute', min_samples_leaf=1, smoothing=1) Target Encode for categorical features. Based on leave one out ...


8

I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it is to category 3). This is definitely not the case for hours or months. However, the issue that you raise is that you want to represent hours and months in a ...


8

Feature engineering that I would consider essential for even tree based algorithms are: Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your model needs to know that something happens on the third Monday of every month, it will be nearly impossible to determine this from timestamps. On a similar vein, ...


8

Q1) Should highly correlated features with the target variable be included or removed from classification and regression problems? Is there a better/elegant explanation to this step? Actually there's no strong reason either to keep or remove features which have a low correlation with the target response, other than reducing the number of features if ...


7

You have time series data which is used to measure the acceleration. You which to identify when the machine is in its nominal state (OFF) and anomalous state (ON). This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem. Preparing you data All of the methods will rely on the ...


7

The issue with building a regression model on all 3 of these is that you are potentially introducing multicollinearity into the model. Although log(input) and sqrt(input) are not linear functions of the input a quick test (using Matlab) shows they are still highly correlated (depending on the range) input=rand(1,100); input_log=log(input); input_rt=sqrt(...


7

"Hand Crafted" features refer to properties derived using various algorithms using the information present in the image itself. For example, two simple features that can be extracted from images are edges and corners. A basic edge detector algorithm works by finding areas where the image intensity "suddenly" changes. To understand that we need to remember ...


7

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect. Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the increased complexity overweight the added value from extra features. ...


7

1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is amenable to learning, but you can always construct features from it that are. The feature you are adding are based on one feature. This is common. However, your ...


7

A note: for those who've ended here looking for a hashing technique, geohash is likely your best choice. Representing latitude and longitude in a single linear scale is not possible due to the fact that their domain is inherently a 3D space. Reducing that as per your needs would require a spatial flattening technique that's unheard of to me. Reasoning As ...


7

Check this post. In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on the nature of the data. Check also this thread. What's the rationale behind it? High cardinality may result in dimensionality curse and actually decrease ...


6

Is this normal? It is not surprising. First, you are using different measures of feature importance. It’s like measuring the importance of people (or simply sorting them) using their a) weight, b) height, c) wealth and d) IQ. With a and b you might get quite similar results, but these results are likely to be different from results obtained with c and d. ...


6

First, transform your columns, then apply linear regression, but do you want to know about the influence of your features on your selected dependent variable? Read this article: http://www.ritchieng.com/machine-learning-evaluate-linear-regression-model/ It provides great insight at how interpret the coefficients given by the algorithm can be interpreted ...


6

If you can keep adding new data (based on a main concept such as area i.e. the ZIP code) and the performance of your model improves, then it is of course allowed... assuming you only care about the final result. There are metrics that will try to guide you with this, such as the Akaike Information Criterion (AIC) or the comparable Bayesian Information ...


6

You basically want to create a column for each product bought, as the presence or absence of each in the list is a feature in itself. See Hadley Wickham’s definition of tidy data. That being said, you seem to have a large number of products. To avoid the course of dimensionality, what I would do is to take your binary bought/not features (or count values ...


Only top voted, non community-wiki answers of a minimum length are eligible