Questions tagged [feature-engineering]
the process of using domain knowledge of the data to create features that improve machine learning algorithms
649
questions
0
votes
1
answer
43
views
Correct order of preprocessing/EDA/feature engineering?
I was wondering if I have the correct order of preprocessing/EDA/feature engineering below?
Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for ...
- 111
0
votes
0
answers
22
views
propper feature encoding
I working with the following data set
also here is it's detailed description of "packet_dat" column
I can't understand how I can encode packet_dat column into proper feature so my ...
- 101
0
votes
1
answer
18
views
Is there a standard data science workflow/decision tree?
I'm looking for some kind of reference that essentially shows an example of an entire data analysis workflow beginning with feature engineering, and ending with analyzing the results.
I know the ...
- 101
1
vote
1
answer
45
views
How do I use ML models to estimate current stress level based on past data?
I am new to machine learning and I cannot understand the difference between estimating current stress level and predicting future stress levels based on historical data. I have been told these are two ...
0
votes
0
answers
14
views
How to lower MSE using polynomial regression?
I have a training dataset with the positions (x and y) of three objects and their velocities at a time t. Then I have a test dataset with the initial positions and a time step x.
The goal is to ...
- 1
0
votes
0
answers
19
views
Preprocessing overheads in Machine Learning
Meta reports that data preprocessing overheads is fast becoming a bottleneck to machine learning training (https://engineering.fb.com/2022/09/19/ml-applications/data-ingestion-machine-learning-...
1
vote
1
answer
25
views
Feature selection for propensity model
I'm trying to build a propensity model for whether or not a customer will buy a second product. I was given data that looks like this:
| Age | Income | DaysSince1stPurchase | Bought2ndProduct |
|:---- ...
- 13
0
votes
0
answers
19
views
Why is the feature direction chosen in the direction associated with largest eigenvalue of $Σ_T$ in case of more than two classes?
Why is the feature direction chosen in the direction associated with largest eigenvalue of $Σ_T$ in case of more than two classes? Please see the following.
- 101
0
votes
0
answers
27
views
Incorporate a new feature or Post-process
Briefly, I am training a model using XGBoost to predict future quantity for the factory to produce. Basic features currently in use are date time features, categories, holiday (binary). I have just ...
- 1
0
votes
0
answers
7
views
Terminology: What is it called when the filter criteria is added to the data table?
When we want to focus on a group within our dataset, it is common to filter it.
Suppose we have a data table showing how many points 4 players earned in a game.
Table 1: Game Scores for 4 Players
Age
...
- 123
0
votes
1
answer
55
views
Unclear points on projection type and selection of distance metric in feature extraction for a set of scenarios
The following is an example from a book (An Introduction to Pattern Recognition and Machine Learning by P. Fieguth, page 85) on feature extraction and selection. Please consider the following figure.
...
0
votes
0
answers
7
views
How to handle multiple values which have multiple subsets?
I am new to datascience. I am trying to predict tab position in msword using features like paragraph text, font name, tab count, tab index etc. In tab position there may be 0 values or multiple ...
0
votes
1
answer
33
views
Is this Dataset somehow skewed?
I am working on a dataset that has 100K points, it's about Customer churn. So I don't know whether this dataset is skewed, incomplete or what. I tried doing some feature engineering on it but couldn't ...
0
votes
0
answers
42
views
One-Hot encoded variables dominates importance among other variables
I am currently training some machine learning models to predict the 28-day compressive strength of cement, a continuous real-valued variable. The available dataset comprises samples from three ...
- 11
0
votes
0
answers
8
views
Grouped Time Restricted Demand Regression with value cap
so I am working on quite an interesting regression task that I haven't encountered before.
Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
0
votes
0
answers
12
views
Optimal method for predicting outcome from many additive, correlated, and sparse features?
Suppose I have many vectors which can take on any of three values, 0, 1, 2. These vectors affect an outcome being predicted, Y. Vectors add together: a vector "A" of the value 2 has twice ...
- 101
0
votes
1
answer
31
views
Best strategy for handling missing groups of features
I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
0
votes
0
answers
15
views
Sending rolling statistics to RNN
I'm curious if anyone has seen cases where sending rolling statistics such as mean, median, min, max, standard deviation, skewness, kurtosis, etc. have been helpful for model accuracy?
If so please ...
- 41
0
votes
0
answers
16
views
Matrix time-series Forecasting with LSTM
I have the following time-series data:
...
0
votes
0
answers
42
views
time series analysis: lag features
I plan to include lag features in my multivariate time series data.
I have 17 input feature and one output feature. both input and output features are time series.
I will insert one more input feature ...
- 1
0
votes
1
answer
31
views
Natural Order should be maintained while ordinal encoding?
I am encoding my ordinal categorical values as VHigh=1, High=2,Med=3,Low=4. Am I doing correct? or order doesn't impact?
If it impacts, how does it impact Decision Tree, Logistic Regression, SVM?
- 11
0
votes
1
answer
32
views
Variable length training data for tabular data neural network regression
I want to predict the age of a parent using the ages of its children.
The problem is that in the data each parent has different numbers of children. How do I create a model that can take variable ...
- 101
-1
votes
1
answer
23
views
How to Data Engineer a dataset to get the best featurres to predict a target class?
In my dataset, I have data of IDs that don't create any meaningful relationship with each other and when I test that dataset on different models I am not getting accuracy more than 40%.
Anyone can ...
0
votes
1
answer
103
views
Feature selection / missing values
What are the top (including new, if any) algorithms to perform features selections without removing or altering the missing data points ? Thanks
- 1
0
votes
1
answer
13
views
feature engineering mechanism
why do we need to rescale some feature having large range I know we do it for faster rate of gradient descent ,but still how does rescaling works? and it doesn't break the model
and does rescaling ...
0
votes
0
answers
31
views
Effect of removing duplicate and identical entries on dimensionality reduction
I have huge data with thousands of observations and millions of features. I need to do clustering so I use PCA/t-SNE/UMAP for dimensionality reduction followed by K-Means.
Currently, I retain only ...
- 101
0
votes
1
answer
81
views
Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value
I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
- 1
1
vote
0
answers
32
views
Density distribution for feature analysis
I trained a ML model on original data with 6373 features, then I trained the same model on compressed data (using autoencoder) and I got an improvement. Finally, I trained the same model on reduced ...
0
votes
2
answers
26
views
How to rank relatedness of two feature in dataset by their distribution?
Let's say we are given a dataset and want to rank them by similarity of distributions. I don't want to use visualization. Is there any sufficient way that you can share with me?
I have an idea like, ...
0
votes
1
answer
93
views
Anomaly Detection: Large number of categories
Looking for some advice.
I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes.
...
- 1
0
votes
1
answer
11
views
How to handle using input feature (clicks) when it is used in target too?
I am trying to create a ranking model, where I am thinking about creating ground truth based on clicks by user. But at same time past clicks made by users seems like a vital input feature too. Any ...
- 1
0
votes
1
answer
128
views
How would I design a database structure for a feature store?
I have a personal project to create predictions for tennis matches.
It currently consists of a Python application and a MySQL database. I extract data from various websites and APIs and store it in ...
- 103
0
votes
0
answers
15
views
Features derived using retrocausality
I have been experimenting with features derived using retrocausality (not to be confused with data leakage) in training models. Are there any examples of prior work in the literature where this form ...
0
votes
0
answers
46
views
FTT Features to use after time-domain is transformed to frequency-domain
Please forgive the question if it sounds trivial/naive, I am from computer science background, not electrical/computer engineering.
I work with GPS trajectory dataset for classification. Data was ...
- 101
0
votes
0
answers
12
views
Feature Engineering and Score Generation for a List of Features
I have a decision tree trained for a binary classification task. It uses a model score plus some other attributes as features. It performs well, majorly because the model score is a strong feature ...
- 21
0
votes
0
answers
33
views
Handling informative NaN values
I was wondering if there is any standard or recommendable way for handling informative NaN values, for example with respect to computing distances.
For informative NaN values, I mean NaN values that ...
- 1
5
votes
2
answers
544
views
How to handle similarity search on mixed data types vectors?
I think this question is one that many beginners run into and I could not find a decent generic guide for it.
My issue is the following. I want to evaluate similarity of vectors which have mixed data ...
- 53
0
votes
1
answer
33
views
How to treat single column with both continuous and categorical data for ML model
I am working on financial data where I have a feature(column) with 90% values between 0-1000 (continuous) and 10% values as -1, -2 and -9. (default values)
Default value definition:
-1: data not ...
- 1
0
votes
0
answers
11
views
How to organize such rich features?
Let me describe the dataset:
250 patients, each patient visited the doctor every 3 months. The minimum number of visits is 4 and the maximum is 17.
For each visit there are peptides that are part of ...
0
votes
0
answers
46
views
Feature generation for multi-label classification
Problem statement
Predict customer's likelihood/propensity to buy multiple category of products from a grocery store give past purchase data (Given a set of users at time 𝑡, predict whether they will ...
- 123
0
votes
1
answer
32
views
important feature selection using dimensionality reduction algorithms
I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
0
votes
0
answers
6
views
Cannot scale after Encoding
After splitting my training and test sets, I successfully encoded my categorical data but whenever I try to Normalize or Standardize the data, it goes back to saying "ValueError: could not ...
0
votes
0
answers
19
views
Feature engineering from HTML & CSS attributes
I have a bunch of HTML files, and inside each there's a particular occurrence of a "substring" which I need to find out. The substring can occur in any tag, and can also have multiple ...
- 101
0
votes
0
answers
33
views
Logistic Regression, Standardization, Stationarity, Differencing
I am going to be using the logistic regression in which I will use L2 Regularization. I have these 4 rolling standard deviation variables. Here are the results of the Augmented Dickey-Fuller Test for ...
0
votes
0
answers
26
views
Why use feature-hashing instead of just remove random words from Bag-of-words?
As far as I understand Feature-Hashing ("Hashing Trick") is that we map some string to an index in an array e.g say we want the resulting dimension of our array to be 5, then maybe the text <...
- 460
0
votes
1
answer
32
views
Feature engineering for interest-based age classification
I have a dataset which has users (rows) with the list of their interests (IABs), which looks like this
...
- 1
0
votes
1
answer
172
views
Which chess notation to feed neural network: FEN or PGN?
I am trying to build a chess AI with a neural network. To learn about how neural networks work and refresh my programming experience. I have some experience with classifiers but not yet with neural ...
- 123
1
vote
1
answer
235
views
Can feature engineering avoid overfitting?
Can feature engineering avoid overfitting? If yes, are there any relevant papers that state this?
- 31
0
votes
1
answer
29
views
How can I know the best number of features to use?
I noticed that developing ml models a very important step in feature engineering is adding new features that can explain better the target variable.
Recently I experienced a situation where by adding ...
0
votes
1
answer
214
views
How to efficiently reduce dimensions of one-hot encoded categorical values?
I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data.
My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
- 13