Questions tagged [feature-engineering]

the process of using domain knowledge of the data to create features that improve machine learning algorithms

Filter by
Sorted by
Tagged with
1 vote
0 answers
6 views

TextVectorization and Autoencoder for feature extraction of text

I'm trying to solve a problem which is as follows: I need to train the autoencoder to extract useful data from text. I will use the trained autoencoder in another model to extract features. The goal ...
0 votes
0 answers
11 views

can someone explain how to create new features using feature interactions?

There is this notebook solving housing prices. https://www.kaggle.com/code/jesucristo/1-house-prices-solution-top-1/notebook?scriptVersionId=12846740 and it had this bit of code, can anyone explain ...
user avatar
  • 1
0 votes
0 answers
10 views

Would I be able to combine features on a different unit scale after normalizing?

I'd like to explore some interactions between my variables but they're on different measurement scales. Would for example the absolute value of the difference of them after scaling make sense? From ...
user avatar
  • 3
0 votes
0 answers
18 views

how to deal with features in pairwaise comparison models?

I am working on a dataset of ATP (Association of Tennis Professionals - men only) tennis games over several years. I want to predict the outcome of tennis so one way to do that is using a Bradley-...
user avatar
1 vote
0 answers
13 views

Discrete wavelet transform - DWT (beginner)

I recently stumbled upon this article : https://www.bportugal.pt/sites/default/files/anexos/papers/wp201612_0.pdf In the paper they use DWT and I am having trouble understanding how to construct them. ...
user avatar
0 votes
2 answers
19 views

do feature selection and model selection must share the same ratio between development set and test set?

As the title, after I performed a Feature Selection, is it mandatory to respect the same ratio (between development set and test set) in Model Selection?
user avatar
0 votes
0 answers
9 views

Can you use gplearn library to improve an SVM model?

I want to know your thoughts on this. Someone on the internet recommended this process to me in order to improve the accuracy of my SVM model: Split dataset with 5 folds stratified k-fold (SKF) Apply ...
user avatar
  • 21
1 vote
0 answers
12 views

Methods for combining instance observations for classification

I am working on a project where I classify tiny moving particles into a few classes (fibers, hairs, glass shards, bubbles). The particles are only a few pixels large and are observed in a few frames ...
user avatar
1 vote
0 answers
19 views

How to deal with feature with different sample size?

I got a dataset that contains 50 features starting from 2009 to 2018. But one of the feature was only availiable since 2015 and unable to recover. I am concerning about if I train a model on the whole ...
user avatar
  • 9
1 vote
1 answer
11 views

Relationship between visualization and feature engineering

Having been in industry for a while, my puzzle on this question still remains unsolved. What exactly is the relationship between visualization and feature engineering? While on Kaggle or elsewhere we ...
user avatar
  • 359
0 votes
0 answers
12 views

Multicolinear Predictors Effect on Model

I know that multicolinear predictors in a model aren't ideal because it causes the model to be sensitive to very minor changes, which then reduces our ability to interpret the effects of each ...
user avatar
1 vote
1 answer
19 views

How to incorporate predictor variable without future information into a model?

I will use an extremely simplified example to ilustrate the question, but I think the answer shsould hold for more generalised cases. Let's say I want to create a time series regression model (the ...
user avatar
  • 11
1 vote
1 answer
15 views

Does feature engineering require absolute accuracy?

Sometimes when I'm studying the datasets, the text field is particularly challenging to handle. For whatever features I want to derive from the text fields, I try to apply some heuristic to ...
user avatar
  • 359
1 vote
1 answer
10 views

Application of bag-of-ngrams in feature engineering of texts

I've got few questions about the application of bag-of-ngrams in feature engineering of texts: How to (or can we?) perform word2vec on bag-of-ngrams? As the feature space of bag of n-gram increases ...
user avatar
  • 359
1 vote
1 answer
26 views

Feature engineering before splitting

This is a sister post to the original closed post (here). Since the data transformation part is done after data spliting on the TRAINING data only, I wonder wouldn't such transformation has dependency ...
user avatar
  • 359
1 vote
1 answer
16 views

Finding contribution/weight of features in output from a series of data

I have a dataset which consists of the following features : a1,a2,.....aN.... sum of all the features together gives a constant output say 100. For eample the dataset may look like this. a1 a2 ...
user avatar
  • 11
3 votes
1 answer
24 views

NLP text representation techniques that preserve word order in sentence?

I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
user avatar
  • 359
1 vote
1 answer
26 views

Do subjective features result to noise?

Suppose I have a dataset containing feature values I don't all agree with, as another person selected them. An example problem could be classifying 'good' or 'bad' music using subjective features, e.g....
user avatar
  • 13
0 votes
1 answer
18 views

How to represent a time duration feature for cases where time is still counting

I have a problem where I am trying to classify the outcome of costumer complaint cases. I have several features already such as type of item bought, reason for complaint etc... I am trying to add a ...
user avatar
0 votes
0 answers
14 views

Clustering by resource use in different categories

Task The task is to cluster units by similar resource use. For example we might have data such as: Resource A Resource B Resource C Resource D Resource E Total Unit 1 500 0 0 0 100 600 Unit 2 0 1,...
user avatar
1 vote
0 answers
15 views

What features/model to use for data with Hysteresis?

I have made a pressure sensor that when graphing Conductance vs Pressure (disregard the actual values in graph), has the following behaviors: First pressed it has one trendline Afterwards when ...
user avatar
  • 111
1 vote
1 answer
18 views

How should I handle time-duration-based columns in classification?

For example, say I am trying to predict whether I will win my next pickleball game. Some features I have are the number of hits, how much water I’ve drinken, etc, and the duration of the match. I’m ...
user avatar
0 votes
0 answers
12 views

How to decide an optimum value for n_features parameter in Sklearn FeatureHasher

I am using hash encoding on a categorical column with 13 different value counts, and ideally speaking one-hot and dummy will give us 12 and 13 columns respectively after encoding. But when it comes to ...
user avatar
1 vote
1 answer
21 views

Lagged Features

Lets look for example, at the forecast the sales of a retail outlet. If I understood the concept correctly, than a lagged feature would be the sales of a previous month t−1. Would it make sense/is it ...
user avatar
1 vote
1 answer
25 views

Feature selection with "overly important" features

I am very new to machine learning modeling, but I encountered a feature selection problem that I hope can get your insights on: For example, I have A,B,C,D as my independent variables and y as my ...
user avatar
0 votes
1 answer
22 views

How to do feature selection or feature engineering in datasets with a lot of features?

To make a good ML model, we have to select features that increase model accuracy and, if needed, to "engineer" features (e.g. apply some function like logarithm or square to linear ...
user avatar
  • 11
1 vote
1 answer
12 views

How To Normalize Feature That Depends Arbitrarily On Date Published

Working on an ML project to predict the number of listens a certain podcast episode of my podcast will get in the first 28 days. The problem is that when I first started recording the podcast would ...
user avatar
1 vote
1 answer
27 views

What to do about a predictor with a feature importance so high above the others that it is the only determining factor in my machine learning model?

I created a logistic regression model with scikit-learn which predicts the outcome of an NFL football game. It predicts the result based on features such as the team's record, opponent's record, pass ...
user avatar
2 votes
1 answer
37 views

How to deal with date features in linear regression?

I need some help about a project. I have a dataframe like that; YEAR MONTH INDICATOR_1 INDICATOR_2 INDICATOR_3 2014 3 0.123 0.495 0.222 My goal is to predict all of the indicator for the next year (...
user avatar
0 votes
0 answers
7 views

How to use macroeconimic data as time series external regressor, when one or more periods at the end are still missing?

In order to find some useful external predictors to boost my time series recipes, I started to have a look at the national statistics office data of Germany. I want to integrate some trade and ...
user avatar
1 vote
1 answer
11 views

How to use a feature recorded in different unit?

I want to use hour as a feature in my random forest model. The challenge that I’m facing is that some observations are recorded based on machine operating hour while others are in engine hour. Without ...
user avatar
1 vote
1 answer
15 views

What to do when you are building a feature and the denominator is zero?

This is something that looks very simple to solve, but I couldn't find any hint - perhaps I'm not asking Google the right question. Let's say you own an Internet Company. You have the total ...
user avatar
0 votes
1 answer
8 views

Is It Okay To Do 0-1 Scaling Then Divide By The Standard Deviation?

If am understanding stuff correctly, if I have a df I can first do 0-1 scaling on it to get equal ranges while preserving the data series's original means and standard deviations and then once I ...
user avatar
0 votes
1 answer
30 views

Does it make sense to add a new calculated column for dates/duration?

I'm using a Random Forest Classifier on some data, and I have two date field, StartDate and EndDate. Does it make sense to ...
user avatar
  • 1,248
1 vote
1 answer
24 views

how to test if labels have actual dependencies on features?

I am trying to train an LSTM(many to one) model with multivariate time series input and a categorical output. after training for quite some time, the resulting model still has low accuracy and high ...
user avatar
0 votes
1 answer
26 views

One-hot encode a numeric categorical feature (e.g. year built, satisfaction out of 10, etc) or not?

I'm trying to understand the pros and cons of different approaches for encoding a certain feature rather than keeping its numerical value. Let's say we have a dataframe that has a Satisfaction column ...
user avatar
  • 3
0 votes
0 answers
30 views

Does the sign of correlation matter in feature selection?

If I understand correctly, the correlation between features and the target can be used to quantify whether those features are relevant to keep, hence the ritual of plotting the correlation matrix as a ...
user avatar
  • 175
0 votes
0 answers
15 views

Different granularity across multiple dataset in ML

I have three datasets about Instagram: One containing comments One containing user info One about user posts The comment table is at one comment per line which can be merged to user info but then in ...
user avatar
  • 83
0 votes
3 answers
108 views

Add timestamp as a feature to model

I am working with time-series data and am interested in adding time-stamp data (as a feature) into the (DNN) model. From the things I have read online so far, my only option is to come up with my own ...
user avatar
  • 85
0 votes
1 answer
9 views

Should I create single feature for each specific word which i find in text or one for all them?

I am doing feature engineering right now for my classification task. In my dataframe I have a column with text messages. I decided to create a binary feature which depends on whether or not in this ...
user avatar
  • 113
5 votes
4 answers
3k views

In ML why selecting the best variables?

Almost all ML notebooks out there have a section where they select the best features to use in the model. Why is this step always there ? How bad can it be to keep a variable that is not correlated ...
user avatar
  • 121
4 votes
1 answer
478 views

Are linear models better when dealing with too many features? If so, why?

I had to build a classification model in order to predict which what would be the user rating by using his/her review. (I was dealing with this dataset: Trip Advisor Hotel Reviews) After some ...
user avatar
  • 151
0 votes
1 answer
36 views

how to do feature engineering on Atari Pong game in code?

In RL context, I know that features are explanatory variables that represent or describe the states. If I want to do feature engineering on atari games and use it to solve RL task, how should I ...
user avatar
  • 101
0 votes
0 answers
16 views

Does every column need to be np.log?

I have questions regarding data cleaning for machine learning. Let's say my dataset has three columns with different skewness For example: label column skewness = 1.500, feature column 1 ...
user avatar
1 vote
1 answer
15 views

Changing the predicted variable from price to price/km due to better visual correlation

I'm working on a dataset of Uber Rides from Kaggle. Of the important variables there are pickup and drop-off coordinates, passenger count, datetime of pickup, distance and the final price. I'm ...
user avatar
0 votes
0 answers
26 views

Aggregation vs binning continuous values

For any generic DNN, When should one aggregate a continuous variable and when should one put them into intervals instead? For example, the number of requests a user has sent in the last N days could ...
user avatar
  • 111
1 vote
1 answer
61 views

Why categorical variable with high cardinality is not preferred but not in numerical variable?

I researched online a bit about categorical variables with high cardinality. Many posts and papers just stop short and conclude that 'it skews model's performance' without going into details why and ...
user avatar
  • 359
0 votes
0 answers
3 views

When would you use feature optimization method instead of exploratory analysis to identify best features?

I have a dataset with around 70 features. I'm currently just plotting graphs and trying to identify key information. I also wish to later do a predictive model. What would be the best way to get the ...
user avatar
0 votes
1 answer
28 views

How Can I Process SageMaker Ground Truth NER JSON Output into DataFrame?

So, I've recently created a job using AWS SageMaker Ground Truth for NER purposes, and have received an output in the form a manifest file. I'm now trying to process the manifest file into a dataframe,...
user avatar
1 vote
0 answers
68 views

Features should be at same order in train and test?

I train ensemble model Logistic regression random forest adaboost class With pcr and random search In pipeline. The features in the train and test are Equal but not in the same order. Will it be ...
user avatar
  • 23

1
2 3 4 5
12