Questions tagged [feature-engineering]
the process of using domain knowledge of the data to create features that improve machine learning algorithms
512
questions
0
votes
0answers
9 views
What is the best programming paradigm for Data Engineering: Object Oriented, Functional, Procedural or a combination of them?
How would you structure a typical Data Engineering pipeline in terms of Programming paradigm? would you keep the traditional procedural approach or use OOP with functional elements? where is a good ...
0
votes
1answer
14 views
Aggregation of low level features for a classifier
The objective is to predict router fail/no fail (1/0) in a future time window with all the data collected over the last hour (i.e. binary target)
The data is received at two different levels:
Router ...
0
votes
0answers
4 views
Should I generalize categorical features if the algorithm handles over-fitting well?
I'm referring to Kaggle feature creation exercise . The data frame contains a column(MSSubClass) that contains these unique values:
...
0
votes
0answers
17 views
Feature Engineering on 3 dimensions data
I'm doing a task where I was given 3 features (a1, a2 and a3) and 3 heavily unbalanced classes. I tried many balancing techniques like SMOTE and undersampling. None of them gives me a reasonable ...
3
votes
1answer
100 views
Finding if an outcome is predictable
Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
0
votes
1answer
15 views
Would it be a good idea to use PCA output as input in models?
I have some dummy variables that indicate the occurrence of an event. There is so many of them, so I used PCA on them, and it appears some of them are rather correlated together.
Would it be a good ...
0
votes
1answer
35 views
Is feature importance in XGBoost or in any other tree based method reliable?
This question is quite long, if you know how feature importance to tree based methods works i suggest you to skip to text below the image.
Feature importance (FI) in tree based methods is given by ...
0
votes
0answers
8 views
How do you aggregate features of lists (pooling alternatives)?
Is it possible to reduce non-correlated multi-dimensional data over features to 1D data?
A working option is pooling (mean/min/max) over an embedding vector (n samples of embeddings of m dimensions). ...
0
votes
0answers
15 views
Year/Month as a feature in Random Forest Classifier
I need to include a Maturity Date feature in my scikit-learn RandomForestClassifier model.
Since the day is too specific, I'm thinking of having a number with the ...
0
votes
0answers
19 views
payment data prediction at test time
I have the payment data of the client. I want to predict the prob of customers paying late with target classes being 0-30 days, 30-60 days, 60-90 days, and 90+ days based on this paper.
The features I ...
1
vote
1answer
14 views
Time Series Target variable taken at much lower sample rate than input features
I have a regression problem that involves predicting a patient's blood pressure from a range of vital sign readings including PTT, PPG, and HR. Each of these input features has been taken at the same ...
0
votes
1answer
22 views
Feature engineering: The more features I add the better RMSE I get?
I have a model with 7 features, I'm trying to figure out if I can improve the performance of this model by adding additional features. So I'm relying on the RMSE to measure the accuracy of my ...
1
vote
1answer
13 views
Traditional alternatives to Caputure Words Sequence information in NLP
What were the traditional/earlier methods in which NLP researchers captured the word sequence information through feature engineering?
I know the current methods which rely on deep learning models ...
1
vote
0answers
26 views
Dataset format for Transformer text-generation
I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...
2
votes
0answers
24 views
Why are correlation matrices used versus a matrix of R^2 values?
I'm relatively new to DS, so forgive me if this is a dumb question or in the wrong forum
When evaluating features it seems that almost everywhere a correlation matrix is used ...
1
vote
0answers
7 views
Does it make more sense to use central tendency methods on training and test sets separately?
I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
2
votes
1answer
21 views
Training & Test feature shape is different from number of columns in dataset
I am making a Sequential Neural Network for regression with 3 dense layers which will be trained on a simple dataset. But before I even get to that part of the code to execute the model I am getting a ...
1
vote
0answers
10 views
Tsallis entropy - advice needed regarding obtaining probability distribution
As is always the way I stumbled across Tsallis entropy on SO whilst looking for something completely different. This soon lead me reading all sorts of interesting but terse academic papers.
I am ...
0
votes
2answers
31 views
XgBoost given targets its only feature but fails when test targets are outside the range of training targets?
I'm learning to use XgBoost, and I'm doing an exercise involving predicting prices. However I'm noticing some weird behavior where XgBoost's predictions deviate from the target value even if I'm ...
2
votes
0answers
18 views
Using on-demand features in machine learning
I have 6 input features $[m1,m2,m3,m4,m5,m6]$.
I am trying to build a model that can predict the value of all 6 of these values using $[m1,m2,m3]$. However, I have the option of asking for another ...
1
vote
1answer
23 views
Encoding a categorical variable with a few billion possible values
I am looking to train a neural network to solve a supervised classification task. But one of my input features is a categorical variable that can have more than a few billion possible values.
For ...
1
vote
1answer
31 views
Does it make sense to use UMAP for dimensionality reduction for modeling (rather then presentation/exploration)?
Reducing dimensionality via PCA before training is a common practice, but PCA cannot makes use of nonlinear relations between features.
I read about UMAP (e.g. https://adanayak.medium.com/...
0
votes
2answers
110 views
Imbalanced classification with bias
The problem:
A business historical heuristic rule for offering a special deal to customers has created a bias in the dataset when trying to use machine learning in order to make a more sophisticated ...
2
votes
2answers
55 views
How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?
I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have ...
0
votes
0answers
16 views
How to encode high dimensional dynamic categorical data?
Example: Facebook newsfeed ranking. This is typically done by relevance scoring each post using a regression model based on a number of features.
Typical features might include information from the ...
1
vote
1answer
30 views
What if outliers still exist after variable transformation?
I have a variable with a skewed distribution.
I applied BoxCox transformation and now the variable follows a Gaussian distribution. But, as seen in the image below in the boxplot, outliers still ...
0
votes
1answer
37 views
A feature highly correlated with target variable
What if one of the predictor variables is highly correlated with the target variable (say 0.9), what should we do? Should we drop it or keep it to build the prediction model(classification or ...
0
votes
2answers
15 views
Is there wights of voice or audio for VGG or Inception?
I want to use VGG16 (or VGG19) for voice clustering task.
I read some articles which suggest to use ...
1
vote
0answers
10 views
spark ml StringIndexer vs OneHotEncoder, when to use which?
Confused as to when to use StringIndexer vs StringIndexer+OneHotEncoder.
The OneHotEncoder docs say
For string type input data, it is common to encode categorical features using StringIndexer first.
...
0
votes
0answers
11 views
Multi-input NN Debugging
I have trained a simple multi-input NN for classification. I have 4 inputs ( one text field & other 3 categorical variables : cat1, cat2 , cat3).
For the text field, I use Glove embeddings in the ...
1
vote
1answer
33 views
Data Leakage when preprocessing categorical features?
I am fairly new to machine learning. I came across the concept of Data Leakage. The article says that always split the data before performing preprocessing steps.
My question is, do steps such as ...
0
votes
2answers
53 views
How do I replace NaN values using group by pivot_table in pandas DataFrame?
I am working on a machine learning practice problem, from https://datahack.analyticsvidhya.com/contest/practice-problem-big-mart-sales-iii/#ProblemStatement
I want to replace the null values in the ...
0
votes
0answers
11 views
In Time Series forecast, should Scaling be done on both train and test features combined ( test is 1 new data point)?
Let say I have a Time series, I'm using sliding/expanding window method to split to train and test data: train would be all the data I have until day x and test is day x+1.
To avoid Data leakage I'm ...
1
vote
1answer
39 views
How to add more weight to certain features?
I have extracted features from two types of signals. Prior to merging them to create one feature vector, I have computed an importance score of every feature within that type of signal.
I would like ...
2
votes
0answers
46 views
Creating a new feature from an existing one using decision trees
Is it possible to create a new feature out of two, or more than two existing features using a decision tree?
If so, how, and can it produce features with good information value that can better help ...
1
vote
0answers
49 views
Anomaly detection for high dimensional categorical data
I have a dataset with around 200+ categorical variables $X_i$ and the sizes of their domain $|X_i|$ range from 2 to 8k. So, if I one-hot encode the combination of these variables, the vector space (...
2
votes
0answers
43 views
Which latent variable model is better to find hidden variable?
Currently, I am exploring the concept of latent variable for regression type datasets.
I have gone through literature of few of ...
1
vote
1answer
26 views
Categorical encoded variables in scikit-learn diabetes dataset [closed]
When using sklearn.datasets import load_diabetes, the sex variable which is categorical, is scaled to continuous values. Is it even legal to scale such variables?
0
votes
1answer
50 views
Data Selection according to Feature Values
I am given a dataset consisting of 10 million molecules. Each row contains:
The average value predicted by an ensemble of regression models trained to predict a
certain property about chemicals (...
1
vote
1answer
31 views
Explanation of random forest performance difference to when using categories and when using dummy variables
I have some hand coded feature which is a category with values "High", "Low", and "Normal".
I created this feature myself and my problem performance (classification) ...
0
votes
0answers
12 views
Classification based on color clustering
I need to classify some domain specific images by analysing their color distribution.
I have annotated data; this last classification step is supervised.
After some preprocessing and masking and other ...
0
votes
0answers
10 views
Do Any Frameworks Provide Better Support for End-To-End Integer-Based Feature Engineering, Modeling, and Inference?
A retail enterprise I work with with wants to switch from its home-grown time series data analysis and prediction system to something more established and with community support. One unique feature ...
0
votes
1answer
16 views
when will the incareful features harm the model?
I am working on financial prediction problem(time-series prediction problem).
I think feature engineering is importance in this problem. So i am careful to check the feature's effectiveness. And i ...
1
vote
1answer
15 views
Standardizing mixed type data
Hope you are doing well. I have some problem with mixed. For the classification problem, can we standardize categorical and numerical variables together or just standardise numerical variable or don't ...
0
votes
2answers
96 views
How to read the labeled enron dataset categories?
I am trying to use the labeled Enron dataset (link) but I am really confused about the labeling system they use. I understand the Cat_[1-12]_level_weight is some ...
0
votes
0answers
16 views
Which are the features selection techniques depending on the combination on cat num columns in independent and dependent features?
I am very confused:
For what I understood I should:
Multicollinearity check with Pearson corr and possibly consider to drop multicolliner features
Then?
I am very confused feature selection should be ...
0
votes
1answer
23 views
How to aggregate features to a group level as a feature in machine learning model?
I am building a model to predict some behavior at a household level. I could roll up income or number of cars etc so that I can take everyone into consideration. But how can I roll up something like ...
1
vote
1answer
41 views
What is the intuition of using clustering for performing feature engineering in machine learning tasks?
I am trying to implement the research paper Combining Boosted Trees with Metafeature Engineering for Predictive Maintenance. The paper has a section called meta feature engineering where they have ...
2
votes
1answer
42 views
How do I combine predictions from classifiers for two different problem?
I am working on a classification problem for predicting whether the shipment is going to be late or not.
I would say the classifier is mediocre at predicting the positive class at the moment. But the ...
0
votes
0answers
29 views
Is my idea of a Feature Store wrong?
Cross-posted on Reddit ML.
Should a Feature Store be part of an enterprise data catalog?
To me, a feature store seems to be a highly niche data catalog but missing a lot of the benefits of having an ...
