Questions tagged [feature-engineering]
The feature-engineering tag has no usage guidance.
1
vote
0answers
7 views
Handling missing values to optimize polynomial features
I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
0answers
13 views
Handle outliers, Losing many data by removing natural outliers
I have 2 skewed features, here is the summary of one of the features
...
1
vote
1answer
16 views
Predicting a cyclic target
I'm familiar with using trigonometric functions to transform cyclic variables for use as features in training a model (most commonly hour of the day or month of the year); I'm now trying to figure out ...
1
vote
2answers
33 views
Feature engineering decrease my cross validation
I'm currently working on a fraud detection data set. I'm evaluating my training data with a 10-skfold roc auc and an estimator of default param LightGBM. But, the problem is every time I try to create ...
0
votes
2answers
18 views
Equivalent of numeric encoding when rows can contain multiple values
If we have a column such as
Name
0 Alice
1 Bob
2 Dave
then, after numeric encoding, it becomes
...
2
votes
0answers
44 views
Beginner Question Related To Data Science Course [closed]
I am Third Year B.Tech student from 3-tier college of India ,Here is no one fellow or collegous who has somebit knowledge about ML or Data science and I am purely sure that I have strong background of ...
1
vote
1answer
19 views
Creating similarity metric with Doc2Vec and additional features
I have a dataset which contains many features. Each record is company that has many features.
For example...
Company A:
Keywords - data, big data, tableau, dashboards, etc.
Industry - Information ...
0
votes
0answers
11 views
Inserting a feature in the training set made of a target variable
I have a timeseries that contain the daily price of items based on the stores , the manufacturers , and other attributes. Other than what we know about not including a signal from target variable in ...
1
vote
0answers
11 views
Suggestions on using model in production 1 test at a time
I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious:
...
3
votes
3answers
65 views
How to handle large number of features in machine learning?
I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case, it's clearly out of question to examine ...
0
votes
2answers
35 views
How to integrate time series data into normal features for machine learning?
I confront a problem where one data source is a "normal" DF with customers as rows (each customer occurs once) and static customer features as columns. The other DF other hand is a big pile of ...
1
vote
1answer
25 views
Imputation missing values other than using Mean, Median in python
I heard that Mean, Median isn't the best way to impute the missing values, why would that be?
In my scenario, I have data like this
...
2
votes
0answers
26 views
Metrics to evaluate features' importance in classification problem (with random forest)
I want to evaluate the importance of each of the features of a 2000x60 dataset in a classification problem with random forest.
The most widely used ones apparrently are:
Cross Entropy-Information ...
3
votes
2answers
22 views
Dummy variable for Categorical values
The question is in reference to solution of Titanic survival predictionat kaggle . As many have did the similar kind of feature extraction,
They have converted some of the numerical features (Age, ...
1
vote
0answers
10 views
Assigning scalar values for PID for order in Neural Network
I have built a neural network using Windows Process's I started off with only two features, the file path with parent process, and the file path with child process. I am slowly adding features for ...
1
vote
1answer
17 views
Using historical label as a feature in my ML model?
I am working on a predictive model to predict change in the price of an asset (up, down, no change). The labeling is based on the derivative of the price and is exponentially smoothed with an alpha of ...
0
votes
0answers
17 views
Adding and Normalizing extra features to Word2Vec representation
My problem is kind of similar to this question
I am currently using a word2vec 100 features representation of my words.
However, I want to add more features to have more similarity between synonyms ...
0
votes
1answer
19 views
kde plot for interpreting the correlation
i have created some new features for my model. I found people use kde plot to find out the correlation between the created feature and the target variable, but I am not really sure how to find the ...
0
votes
0answers
15 views
How to understand when partial dependence plot and feature importance don't agree
I'm checking partial dependence plot and feature importance on my binary classification using gradient boosting.
The top feature based on the feature importance is a flat line at partial dependence ...
1
vote
2answers
26 views
how to represent location-code as a feature in machine learning model?
I am trying to predict the damage to a buildings after earthquake on a dataset which contains "district number" as feature. I think the feature will have a significant importance in predicting the ...
1
vote
0answers
35 views
How to do feature engineering for email cleaning / text extraction?
I have a large batch of email data that I want to analyse. In order to do that, I need to first prepare the data, as the messages are quite often >80% noise. Generally speaking, my dataset's structure ...
0
votes
1answer
71 views
Categorical data for sklearns Isolation Forrest
I'm trying to do anomaly detection with Isolation Forests (IF) in sklearn.
Except for the fact that it is a great method of anomaly detection, I also want to use it because about half of my features ...
0
votes
2answers
22 views
How to handle NaNs for ratio feature for binary classifier?
I'm creating a churn model and would like to create a ratio (# customers / total transaction) for each merchant. About 70% of the data are NaNs (zero/zero).
I was wondering what I should impute for ...
0
votes
2answers
58 views
What is a good approach for a lifespan?
Let's say I wan't to predict the lifespan of an ad in a listing.
I know a bunch of thing from the ad like:
the title
the price
the location
etc
The target value is the duration of the ad in the ...
1
vote
2answers
32 views
How would you deal with inf. or NA for rate or ratio as a feature variable
I'm trying to create a feature for a churn model (binary classifier).
The feature is mean of sales growth rates for several months. But if I just take the mean of sales for several months, I often ...
0
votes
1answer
16 views
automatic feature selection
I have a lot (thousands are possible) of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the ...
1
vote
2answers
27 views
How to standarize feature vector with data in different scales?
Let's suppose I have a dataset with numerical attributes of different types.
Let's suppose I want to employ a Neural Network for supervised classification with that dataset. For that, I need to ...
1
vote
0answers
21 views
Classifying variable types on a list of variables
I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
0
votes
1answer
15 views
How to transform dictionary data into a string vector?
I have key,value data where each record is in a Python string.
An example record looks like this:
...
0
votes
1answer
15 views
How to aggregate data where instances occur over different time intervals [closed]
I am working on a problem in which I have several instances that have predictors that have activity over various different time periods (i.e. <3 months to well over 20 months.) Originally I ...
3
votes
0answers
50 views
Extract features from a survey
I need to use the answers from a questionnaire for training a classifier.
I discovered that some questions can have nested sub-questions..
Let's say (just an example) that I want to predict whether a ...
0
votes
0answers
20 views
Does it make sense to add word embeddings as additional features for LSTM model?
I have an LSTM model. This model takes as input tokens. Those tokens represent XML markups extracted from some XML files. My model is working fine. However, I want to optimize it by adding word ...
2
votes
1answer
37 views
How to model a Machine learning problem considering links between features
Context:
To predict employee turnover ( will an employee leave? ), I have used one of the classification algorithms (LDA) to train my dataset, and then make predictions.
The dataset is quite small (...
2
votes
2answers
53 views
Time series binary classificaiton with labelling issues
My situation is quite complicated so I will give a similar example from a simpler domain. Suppose we want to try to predict WHEN a mobile game users will make a purchase if given a sale. Almost every ...
0
votes
0answers
17 views
Should I rescale tfidf features?
I have a dataset which contains both text and numeric features.
I have encoded the text ones using the TfidfVectorizer from sklearn.
I would now like to apply logistic regression to the resulting ...
0
votes
0answers
8 views
About effective features for cut out images
What you want to ask
We are looking for valid features (regardless of local and global) for data extracted from arbitrary areas from images.
Ultimately, I think that it can be applied to matching of ...
0
votes
0answers
10 views
Is it bad to add bias to the model?
I have this data where I have to predict the probability of a rare event, and there is a feature that always precedes the event when it turns negative. However, the magnitude and variance of the ...
3
votes
1answer
225 views
Too much inputs = overfitting?
First question : can I mix different sorts of inputs types for example, height and age (of course my inputs are normalized)? in general, can we mix different types of inputs in a neural network ?
...
9
votes
3answers
620 views
Is this a good practice of feature engineering?
I have a practical question about feature engineering... say I want to predict house prices by using logistic regression and used a bunch of features including zip code. Then by checking the feature ...
0
votes
2answers
42 views
Training a regression algorithm with a variable number of features
I need to train a regression algorithm with multiple features and a single label (predicted value). The problem is that this algorithm has to be able to do on-line learning and the number of features ...
1
vote
1answer
29 views
feature engineering in test and train sets (on combined data or separately on train and test)
Background: As part of prediction analysis, I am given a train and test dataset. Both train and test data have numerical and categorical predictor variables and additionally, train data has a ...
0
votes
1answer
18 views
Devise a single metric to measure retention of listeners using the service on a daily basis for the last month
I have dummy data for a music streaming service, the data is as follows:-
Monthly uniques = the number of unique listeners on each platform for the entirety of a given month
All-time uniques = the ...
-1
votes
1answer
20 views
Is it safe to say if features are generated once for a dataset, it may be used for any relevant algorithm?
If I have generated features using state of the art feature engineering methods of a dataset, can I use it for any kind of algorithm to build the model apart from few modifications in the features so ...
0
votes
1answer
24 views
Storing engineered features in a database
I have some data in raw csv files which I would like to store in a MySQL database. The problem is there are constant feature engineering done on this dataset so coming up with one schema to fit all ...
1
vote
3answers
294 views
Time series feature extraction from raw sensor data for classification?
I have a tabular raw data from sensors with associated label and i want to extract the time series features like mean,max,min and std from the data all the sensor data and form another table or export ...
1
vote
0answers
23 views
Feature Engineering for POS register events: Anomaly Detection
I am working on a dataset with only one variable: POS Journal Events.It has different values such as Items ordered, Order Placed, OrderItems,Discount applied, Promotion applied, order voided etc. I ...
2
votes
0answers
21 views
How to combine heterogeneous image features extracted with different algorithms for similar image retrieval?
Say I have access to several pre-trained CNNs (e.g. AlexNet, VGG, GoogleLeNet, ResNet, DenseNet, etc.) which I can use to extract features from an image by saving the activations of some hidden layer ...
1
vote
0answers
12 views
Feature building - phone device type
I want to use phone type (e.g., "iPhone 6s", "Lenovo K8 Note", etc.) as a feature in a model. Does anyone know of a way to obtain release date, price, etc of a phone (besides scraping)? That seems ...
3
votes
1answer
127 views
Should one hot vectors be scaled with numerical attributes
In the case of having a combination of categorical and numerical Attributes, I usually convert the categorical attributes to one hot vectors. My question is do I leave those vectors as is and scale ...
2
votes
0answers
49 views
Exploratory analysis and feature engineering for time till failure prediction using sensor data of engines
I am trying to do some data exploration and analysis on a dataset of engine sensor readings. I would like to determine if the data I have is good enough to predict a time till failure and possibly ...
