Questions tagged [feature-engineering]

the process of using domain knowledge of the data to create features that improve machine learning algorithms

Filter by
Sorted by
Tagged with
0 votes
1 answer
43 views

Correct order of preprocessing/EDA/feature engineering?

I was wondering if I have the correct order of preprocessing/EDA/feature engineering below? Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for ...
Katsu's user avatar
  • 111
0 votes
0 answers
22 views

propper feature encoding

I working with the following data set also here is it's detailed description of "packet_dat" column I can't understand how I can encode packet_dat column into proper feature so my ...
Roma's user avatar
  • 101
0 votes
1 answer
18 views

Is there a standard data science workflow/decision tree?

I'm looking for some kind of reference that essentially shows an example of an entire data analysis workflow beginning with feature engineering, and ending with analyzing the results. I know the ...
David's user avatar
  • 101
1 vote
1 answer
45 views

How do I use ML models to estimate current stress level based on past data?

I am new to machine learning and I cannot understand the difference between estimating current stress level and predicting future stress levels based on historical data. I have been told these are two ...
user123456789's user avatar
0 votes
0 answers
14 views

How to lower MSE using polynomial regression?

I have a training dataset with the positions (x and y) of three objects and their velocities at a time t. Then I have a test dataset with the initial positions and a time step x. The goal is to ...
iknownas's user avatar
0 votes
0 answers
19 views

Preprocessing overheads in Machine Learning

Meta reports that data preprocessing overheads is fast becoming a bottleneck to machine learning training (https://engineering.fb.com/2022/09/19/ml-applications/data-ingestion-machine-learning-...
Rajath Shashidhara's user avatar
1 vote
1 answer
25 views

Feature selection for propensity model

I'm trying to build a propensity model for whether or not a customer will buy a second product. I was given data that looks like this: | Age | Income | DaysSince1stPurchase | Bought2ndProduct | |:---- ...
BlueSkyz's user avatar
0 votes
0 answers
19 views

Why is the feature direction chosen in the direction associated with largest eigenvalue of $Σ_T$ in case of more than two classes?

Why is the feature direction chosen in the direction associated with largest eigenvalue of $Σ_T$ in case of more than two classes? Please see the following.
DSPinfinity's user avatar
0 votes
0 answers
27 views

Incorporate a new feature or Post-process

Briefly, I am training a model using XGBoost to predict future quantity for the factory to produce. Basic features currently in use are date time features, categories, holiday (binary). I have just ...
Bourbon's user avatar
0 votes
0 answers
7 views

Terminology: What is it called when the filter criteria is added to the data table?

When we want to focus on a group within our dataset, it is common to filter it. Suppose we have a data table showing how many points 4 players earned in a game. Table 1: Game Scores for 4 Players Age ...
madprogramer's user avatar
0 votes
1 answer
55 views

Unclear points on projection type and selection of distance metric in feature extraction for a set of scenarios

The following is an example from a book (An Introduction to Pattern Recognition and Machine Learning by P. Fieguth, page 85) on feature extraction and selection. Please consider the following figure. ...
user154739's user avatar
0 votes
0 answers
7 views

How to handle multiple values which have multiple subsets?

I am new to datascience. I am trying to predict tab position in msword using features like paragraph text, font name, tab count, tab index etc. In tab position there may be 0 values or multiple ...
Kamal Budhathoki's user avatar
0 votes
1 answer
33 views

Is this Dataset somehow skewed?

I am working on a dataset that has 100K points, it's about Customer churn. So I don't know whether this dataset is skewed, incomplete or what. I tried doing some feature engineering on it but couldn't ...
Harshal R's user avatar
0 votes
0 answers
42 views

One-Hot encoded variables dominates importance among other variables

I am currently training some machine learning models to predict the 28-day compressive strength of cement, a continuous real-valued variable. The available dataset comprises samples from three ...
Felipe's user avatar
  • 11
0 votes
0 answers
8 views

Grouped Time Restricted Demand Regression with value cap

so I am working on quite an interesting regression task that I haven't encountered before. Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
Martin Pichler's user avatar
0 votes
0 answers
12 views

Optimal method for predicting outcome from many additive, correlated, and sparse features?

Suppose I have many vectors which can take on any of three values, 0, 1, 2. These vectors affect an outcome being predicted, Y. Vectors add together: a vector "A" of the value 2 has twice ...
BigMistake's user avatar
0 votes
1 answer
31 views

Best strategy for handling missing groups of features

I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
recentadvances's user avatar
0 votes
0 answers
16 views

Sending rolling statistics to RNN

I'm curious if anyone has seen cases where sending rolling statistics such as mean, median, min, max, standard deviation, skewness, kurtosis, etc. have been helpful for model accuracy? If so please ...
noNameTed's user avatar
0 votes
0 answers
16 views

Matrix time-series Forecasting with LSTM

I have the following time-series data: ...
Louis GRIMALDI's user avatar
0 votes
0 answers
42 views

time series analysis: lag features

I plan to include lag features in my multivariate time series data. I have 17 input feature and one output feature. both input and output features are time series. I will insert one more input feature ...
mehmet's user avatar
  • 1
0 votes
1 answer
31 views

Natural Order should be maintained while ordinal encoding?

I am encoding my ordinal categorical values as VHigh=1, High=2,Med=3,Low=4. Am I doing correct? or order doesn't impact? If it impacts, how does it impact Decision Tree, Logistic Regression, SVM?
Pramod yadav's user avatar
0 votes
1 answer
32 views

Variable length training data for tabular data neural network regression

I want to predict the age of a parent using the ages of its children. The problem is that in the data each parent has different numbers of children. How do I create a model that can take variable ...
Pibben's user avatar
  • 101
-1 votes
1 answer
23 views

How to Data Engineer a dataset to get the best featurres to predict a target class?

In my dataset, I have data of IDs that don't create any meaningful relationship with each other and when I test that dataset on different models I am not getting accuracy more than 40%. Anyone can ...
Farhan Aslam's user avatar
0 votes
1 answer
105 views

Feature selection / missing values

What are the top (including new, if any) algorithms to perform features selections without removing or altering the missing data points ? Thanks
Zak's user avatar
  • 1
0 votes
1 answer
13 views

feature engineering mechanism

why do we need to rescale some feature having large range I know we do it for faster rate of gradient descent ,but still how does rescaling works? and it doesn't break the model and does rescaling ...
rushi jhala's user avatar
0 votes
0 answers
32 views

Effect of removing duplicate and identical entries on dimensionality reduction

I have huge data with thousands of observations and millions of features. I need to do clustering so I use PCA/t-SNE/UMAP for dimensionality reduction followed by K-Means. Currently, I retain only ...
Quiescent's user avatar
  • 101
0 votes
1 answer
81 views

Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value

I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
vito97's user avatar
  • 1
1 vote
0 answers
32 views

Density distribution for feature analysis

I trained a ML model on original data with 6373 features, then I trained the same model on compressed data (using autoencoder) and I got an improvement. Finally, I trained the same model on reduced ...
Skander Hamdi's user avatar
0 votes
2 answers
26 views

How to rank relatedness of two feature in dataset by their distribution?

Let's say we are given a dataset and want to rank them by similarity of distributions. I don't want to use visualization. Is there any sufficient way that you can share with me? I have an idea like, ...
Ibrahim Rustamov's user avatar
0 votes
1 answer
94 views

Anomaly Detection: Large number of categories

Looking for some advice. I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes. ...
darren's user avatar
  • 1
0 votes
1 answer
11 views

How to handle using input feature (clicks) when it is used in target too?

I am trying to create a ranking model, where I am thinking about creating ground truth based on clicks by user. But at same time past clicks made by users seems like a vital input feature too. Any ...
dcusmeb's user avatar
0 votes
1 answer
128 views

How would I design a database structure for a feature store?

I have a personal project to create predictions for tennis matches. It currently consists of a Python application and a MySQL database. I extract data from various websites and APIs and store it in ...
Jossy's user avatar
  • 103
0 votes
0 answers
15 views

Features derived using retrocausality

I have been experimenting with features derived using retrocausality (not to be confused with data leakage) in training models. Are there any examples of prior work in the literature where this form ...
Adam Patterson's user avatar
0 votes
0 answers
48 views

FTT Features to use after time-domain is transformed to frequency-domain

Please forgive the question if it sounds trivial/naive, I am from computer science background, not electrical/computer engineering. I work with GPS trajectory dataset for classification. Data was ...
Amina Umar's user avatar
0 votes
0 answers
12 views

Feature Engineering and Score Generation for a List of Features

I have a decision tree trained for a binary classification task. It uses a model score plus some other attributes as features. It performs well, majorly because the model score is a strong feature ...
szheng's user avatar
  • 21
0 votes
0 answers
33 views

Handling informative NaN values

I was wondering if there is any standard or recommendable way for handling informative NaN values, for example with respect to computing distances. For informative NaN values, I mean NaN values that ...
fbaroni's user avatar
5 votes
2 answers
551 views

How to handle similarity search on mixed data types vectors?

I think this question is one that many beginners run into and I could not find a decent generic guide for it. My issue is the following. I want to evaluate similarity of vectors which have mixed data ...
Chapo's user avatar
  • 53
0 votes
1 answer
33 views

How to treat single column with both continuous and categorical data for ML model

I am working on financial data where I have a feature(column) with 90% values between 0-1000 (continuous) and 10% values as -1, -2 and -9. (default values) Default value definition: -1: data not ...
Advin's user avatar
  • 1
0 votes
0 answers
11 views

How to organize such rich features?

Let me describe the dataset: 250 patients, each patient visited the doctor every 3 months. The minimum number of visits is 4 and the maximum is 17. For each visit there are peptides that are part of ...
Jonathan Oren's user avatar
0 votes
0 answers
46 views

Feature generation for multi-label classification

Problem statement Predict customer's likelihood/propensity to buy multiple category of products from a grocery store give past purchase data (Given a set of users at time 𝑡, predict whether they will ...
cryp's user avatar
  • 123
0 votes
1 answer
34 views

important feature selection using dimensionality reduction algorithms

I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
Gajanan Kothawade's user avatar
0 votes
0 answers
6 views

Cannot scale after Encoding

After splitting my training and test sets, I successfully encoded my categorical data but whenever I try to Normalize or Standardize the data, it goes back to saying "ValueError: could not ...
olashile adeleye's user avatar
0 votes
0 answers
19 views

Feature engineering from HTML & CSS attributes

I have a bunch of HTML files, and inside each there's a particular occurrence of a "substring" which I need to find out. The substring can occur in any tag, and can also have multiple ...
Tharsalys's user avatar
  • 101
0 votes
0 answers
33 views

Logistic Regression, Standardization, Stationarity, Differencing

I am going to be using the logistic regression in which I will use L2 Regularization. I have these 4 rolling standard deviation variables. Here are the results of the Augmented Dickey-Fuller Test for ...
DomIsAwesomee's user avatar
0 votes
0 answers
26 views

Why use feature-hashing instead of just remove random words from Bag-of-words?

As far as I understand Feature-Hashing ("Hashing Trick") is that we map some string to an index in an array e.g say we want the resulting dimension of our array to be 5, then maybe the text <...
CutePoison's user avatar
0 votes
1 answer
32 views

Feature engineering for interest-based age classification

I have a dataset which has users (rows) with the list of their interests (IABs), which looks like this ...
theodre7's user avatar
0 votes
1 answer
173 views

Which chess notation to feed neural network: FEN or PGN?

I am trying to build a chess AI with a neural network. To learn about how neural networks work and refresh my programming experience. I have some experience with classifiers but not yet with neural ...
NG.'s user avatar
  • 123
1 vote
1 answer
237 views

Can feature engineering avoid overfitting?

Can feature engineering avoid overfitting? If yes, are there any relevant papers that state this?
stack offer's user avatar
0 votes
1 answer
29 views

How can I know the best number of features to use?

I noticed that developing ml models a very important step in feature engineering is adding new features that can explain better the target variable. Recently I experienced a situation where by adding ...
Flavio Brienza's user avatar
0 votes
1 answer
214 views

How to efficiently reduce dimensions of one-hot encoded categorical values?

I'm currently working on a project where I'm using an LSTM to learn and predict sequences of categorical data. My dataset consists of variable-length sequences of items $s_i = [x_{i_0}, x_{i_1}, ..., ...
acxcv's user avatar
  • 13

1
2 3 4 5
13