Questions tagged [feature-engineering]
the process of using domain knowledge of the data to create features that improve machine learning algorithms
575
questions
1
vote
0
answers
6
views
TextVectorization and Autoencoder for feature extraction of text
I'm trying to solve a problem which is as follows:
I need to train the autoencoder to extract useful data from text. I will use the trained autoencoder in another model to extract features.
The goal ...
0
votes
0
answers
11
views
can someone explain how to create new features using feature interactions?
There is this notebook solving housing prices.
https://www.kaggle.com/code/jesucristo/1-house-prices-solution-top-1/notebook?scriptVersionId=12846740
and it had this bit of code, can anyone explain ...
- 1
0
votes
0
answers
10
views
Would I be able to combine features on a different unit scale after normalizing?
I'd like to explore some interactions between my variables but they're on different measurement scales.
Would for example the absolute value of the difference of them after scaling make sense?
From ...
- 3
0
votes
0
answers
18
views
how to deal with features in pairwaise comparison models?
I am working on a dataset of ATP (Association of Tennis Professionals - men only) tennis games
over several years. I want to predict the outcome of tennis so one way to do that is using a Bradley-...
1
vote
0
answers
13
views
Discrete wavelet transform - DWT (beginner)
I recently stumbled upon this article : https://www.bportugal.pt/sites/default/files/anexos/papers/wp201612_0.pdf
In the paper they use DWT and I am having trouble understanding how to construct them. ...
- 106
0
votes
2
answers
19
views
do feature selection and model selection must share the same ratio between development set and test set?
As the title, after I performed a Feature Selection, is it mandatory to respect the same ratio (between development set and test set) in Model Selection?
0
votes
0
answers
9
views
Can you use gplearn library to improve an SVM model?
I want to know your thoughts on this.
Someone on the internet recommended this process to me in order to improve the accuracy of my SVM model:
Split dataset with 5 folds stratified k-fold (SKF)
Apply ...
- 21
1
vote
0
answers
12
views
Methods for combining instance observations for classification
I am working on a project where I classify tiny moving particles into a few classes (fibers, hairs, glass shards, bubbles). The particles are only a few pixels large and are observed in a few frames ...
1
vote
0
answers
19
views
How to deal with feature with different sample size?
I got a dataset that contains 50 features starting from 2009 to 2018. But one of the feature was only availiable since 2015 and unable to recover. I am concerning about if I train a model on the whole ...
- 9
1
vote
1
answer
11
views
Relationship between visualization and feature engineering
Having been in industry for a while, my puzzle on this question still remains unsolved. What exactly is the relationship between visualization and feature engineering? While on Kaggle or elsewhere we ...
- 359
0
votes
0
answers
12
views
Multicolinear Predictors Effect on Model
I know that multicolinear predictors in a model aren't ideal because it causes the model to be sensitive to very minor changes, which then reduces our ability to interpret the effects of each ...
- 13
1
vote
1
answer
19
views
How to incorporate predictor variable without future information into a model?
I will use an extremely simplified example to ilustrate the question, but I think the answer shsould hold for more generalised cases.
Let's say I want to create a time series regression model (the ...
- 11
1
vote
1
answer
15
views
Does feature engineering require absolute accuracy?
Sometimes when I'm studying the datasets, the text field is particularly challenging to handle. For whatever features I want to derive from the text fields, I try to apply some heuristic to ...
- 359
1
vote
1
answer
10
views
Application of bag-of-ngrams in feature engineering of texts
I've got few questions about the application of bag-of-ngrams in feature engineering of texts:
How to (or can we?) perform word2vec on bag-of-ngrams?
As the feature space of bag of n-gram increases ...
- 359
1
vote
1
answer
26
views
Feature engineering before splitting
This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
- 359
1
vote
1
answer
16
views
Finding contribution/weight of features in output from a series of data
I have a dataset which consists of the following features :
a1,a2,.....aN.... sum of all the features together gives a constant output say 100. For eample the dataset may look like this.
a1 a2 ...
- 11
3
votes
1
answer
24
views
NLP text representation techniques that preserve word order in sentence?
I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
- 359
1
vote
1
answer
26
views
Do subjective features result to noise?
Suppose I have a dataset containing feature values I don't all agree with, as another person selected them. An example problem could be classifying 'good' or 'bad' music using subjective features, e.g....
- 13
0
votes
1
answer
18
views
How to represent a time duration feature for cases where time is still counting
I have a problem where I am trying to classify the outcome of costumer complaint cases. I have several features already such as type of item bought, reason for complaint etc...
I am trying to add a ...
- 3
0
votes
0
answers
14
views
Clustering by resource use in different categories
Task
The task is to cluster units by similar resource use.
For example we might have data such as:
Resource A
Resource B
Resource C
Resource D
Resource E
Total
Unit 1
500
0
0
0
100
600
Unit 2
0
1,...
1
vote
0
answers
15
views
What features/model to use for data with Hysteresis?
I have made a pressure sensor that when graphing Conductance vs Pressure (disregard the actual values in graph), has the following behaviors:
First pressed it has one trendline
Afterwards when ...
- 111
1
vote
1
answer
18
views
How should I handle time-duration-based columns in classification?
For example, say I am trying to predict whether I will win my next pickleball game. Some features I have are the number of hits, how much water I’ve drinken, etc, and the duration of the match.
I’m ...
- 111
0
votes
0
answers
12
views
How to decide an optimum value for n_features parameter in Sklearn FeatureHasher
I am using hash encoding on a categorical column with 13 different value counts, and ideally speaking one-hot and dummy will give us 12 and 13 columns respectively after encoding. But when it comes to ...
1
vote
1
answer
21
views
Lagged Features
Lets look for example, at the forecast the sales of a retail outlet.
If I understood the concept correctly, than a lagged feature would be the sales of a previous month t−1.
Would it make sense/is it ...
1
vote
1
answer
25
views
Feature selection with "overly important" features
I am very new to machine learning modeling, but I encountered a feature selection problem that I hope can get your insights on:
For example, I have A,B,C,D as my independent variables and y as my
...
- 13
0
votes
1
answer
22
views
How to do feature selection or feature engineering in datasets with a lot of features?
To make a good ML model, we have to select features that increase model accuracy and, if needed, to "engineer" features (e.g. apply some function like logarithm or square to linear ...
- 11
1
vote
1
answer
12
views
How To Normalize Feature That Depends Arbitrarily On Date Published
Working on an ML project to predict the number of listens a certain podcast episode of my podcast will get in the first 28 days.
The problem is that when I first started recording the podcast would ...
- 113
1
vote
1
answer
27
views
What to do about a predictor with a feature importance so high above the others that it is the only determining factor in my machine learning model?
I created a logistic regression model with scikit-learn which predicts the outcome of an NFL football game. It predicts the result based on features such as the team's record, opponent's record, pass ...
2
votes
1
answer
37
views
How to deal with date features in linear regression?
I need some help about a project.
I have a dataframe like that;
YEAR
MONTH
INDICATOR_1
INDICATOR_2
INDICATOR_3
2014
3
0.123
0.495
0.222
My goal is to predict all of the indicator for the next year (...
- 21
0
votes
0
answers
7
views
How to use macroeconimic data as time series external regressor, when one or more periods at the end are still missing?
In order to find some useful external predictors to boost my time series recipes, I started to have a look at the national statistics office data of Germany.
I want to integrate some trade and ...
- 143
1
vote
1
answer
11
views
How to use a feature recorded in different unit?
I want to use hour as a feature in my random forest model. The challenge that I’m facing is that some observations are recorded based on machine operating hour while others are in engine hour. Without ...
- 11
1
vote
1
answer
15
views
What to do when you are building a feature and the denominator is zero?
This is something that looks very simple to solve, but I couldn't find any hint - perhaps I'm not asking Google the right question.
Let's say you own an Internet Company. You have the total ...
- 11
0
votes
1
answer
8
views
Is It Okay To Do 0-1 Scaling Then Divide By The Standard Deviation?
If am understanding stuff correctly, if I have a df I can first do 0-1 scaling on it to get equal ranges while preserving the data series's original means and standard deviations and then once I ...
- 11
0
votes
1
answer
30
views
Does it make sense to add a new calculated column for dates/duration?
I'm using a Random Forest Classifier on some data, and I have two date field, StartDate and EndDate. Does it make sense to ...
- 1,248
1
vote
1
answer
24
views
how to test if labels have actual dependencies on features?
I am trying to train an LSTM(many to one) model with multivariate time series input and a categorical output.
after training for quite some time, the resulting model still has low accuracy and high ...
- 11
0
votes
1
answer
26
views
One-hot encode a numeric categorical feature (e.g. year built, satisfaction out of 10, etc) or not?
I'm trying to understand the pros and cons of different approaches for encoding a certain feature rather than keeping its numerical value.
Let's say we have a dataframe that has a Satisfaction column ...
- 3
0
votes
0
answers
30
views
Does the sign of correlation matter in feature selection?
If I understand correctly, the correlation between features and the target can be used to quantify whether those features are relevant to keep, hence the ritual of plotting the correlation matrix as a ...
- 175
0
votes
0
answers
15
views
Different granularity across multiple dataset in ML
I have three datasets about Instagram:
One containing comments
One containing user info
One about user posts
The comment table is at one comment per line which can be merged to user info but then in ...
- 83
0
votes
3
answers
108
views
Add timestamp as a feature to model
I am working with time-series data and am interested in adding time-stamp data (as a feature) into the (DNN) model. From the things I have read online so far, my only option is to come up with my own ...
- 85
0
votes
1
answer
9
views
Should I create single feature for each specific word which i find in text or one for all them?
I am doing feature engineering right now for my classification task. In my dataframe I have a column with text messages. I decided to create a binary feature which depends on whether or not in this ...
- 113
5
votes
4
answers
3k
views
In ML why selecting the best variables?
Almost all ML notebooks out there have a section where they select the best features to use in the model. Why is this step always there ? How bad can it be to keep a variable that is not correlated ...
- 121
4
votes
1
answer
478
views
Are linear models better when dealing with too many features? If so, why?
I had to build a classification model in order to predict which what would be the user rating by using his/her review. (I was dealing with this dataset: Trip Advisor Hotel Reviews)
After some ...
- 151
0
votes
1
answer
36
views
how to do feature engineering on Atari Pong game in code?
In RL context, I know that features are explanatory variables that represent or describe the states. If I want to do feature engineering on atari games and use it to solve RL task, how should I ...
- 101
0
votes
0
answers
16
views
Does every column need to be np.log?
I have questions regarding data cleaning for machine learning.
Let's say my dataset has three columns with different skewness
For example:
label column skewness = 1.500, feature column 1 ...
1
vote
1
answer
15
views
Changing the predicted variable from price to price/km due to better visual correlation
I'm working on a dataset of Uber Rides from Kaggle. Of the important variables there are pickup and drop-off coordinates, passenger count, datetime of pickup, distance and the final price. I'm ...
0
votes
0
answers
26
views
Aggregation vs binning continuous values
For any generic DNN, When should one aggregate a continuous variable and when should one put them into intervals instead? For example, the number of requests a user has sent in the last N days could ...
- 111
1
vote
1
answer
61
views
Why categorical variable with high cardinality is not preferred but not in numerical variable?
I researched online a bit about categorical variables with high cardinality. Many posts and papers just stop short and conclude that 'it skews model's performance' without going into details why and ...
- 359
0
votes
0
answers
3
views
When would you use feature optimization method instead of exploratory analysis to identify best features?
I have a dataset with around 70 features. I'm currently just plotting graphs and trying to identify key information. I also wish to later do a predictive model.
What would be the best way to get the ...
0
votes
1
answer
28
views
How Can I Process SageMaker Ground Truth NER JSON Output into DataFrame?
So, I've recently created a job using AWS SageMaker Ground Truth for NER purposes, and have received an output in the form a manifest file. I'm now trying to process the manifest file into a dataframe,...
1
vote
0
answers
68
views
Features should be at same order in train and test?
I train ensemble model
Logistic regression
random forest
adaboost class
With pcr and random search
In pipeline.
The features in the train and test are Equal but not in the same order.
Will it be ...
- 23