Questions tagged [feature-engineering]

the process of using domain knowledge of the data to create features that improve machine learning algorithms

Filter by
Sorted by
Tagged with
1
vote
0answers
10 views

vertical or horizontal storage of timesteps in feature store

I'd like to use a feature store to store some time series and I asked myself what's the best way to store the timesteps. Is it better to store each timestep horizontal and then doing windowing after ...
0
votes
1answer
17 views

Are there any search algorithms for feature optimization similar to RFE, but which consider all possible combinations?

Does anyone know any good search algorithms for feature optimization that search through every possible combination to find the optimal combination of features for maximum predictive power? (...
1
vote
0answers
26 views

What is the best way to group similar columns in a dataset

I have a datasets with many columns (from 16 to 2500 columns) I have built a similarity function that rates the percentage of how similar these columns are. Example: ...
0
votes
1answer
23 views

Outlier treatment

I am working on a regression problem where I have a lot of outliers in multiple variables. As far as I can think of, there are 3 things I can do to outliers. Remove them (least attractive option) ...
1
vote
1answer
26 views

Feature engineering using XGBoost regressor [duplicate]

If I want to train a regression model through tree based algorithms like XGBoost. Suppose that there have 5 features x1, x2, x3, x4, x5 and a target y. And some experts said x2 minus x3 is highly ...
0
votes
1answer
33 views

Should one log transform discrete numerical variables?

I am working on a Linear Regression problem and one of the assumptions of a Linear Regression model is that the features should be Normally Distributed. Hence to convert my non linear features to ...
0
votes
0answers
16 views

Feature engineering with "time since last purchase" and discounting the feature. (Rstudio)

I was not sure if I should post this on cross-validated, stackoverflow or here. But please correct me if this is the wrong forum for this question. I have the following dataset: ...
1
vote
0answers
13 views

Feature creation: Problem with correlated features?

I recently started to read about feature creation. I've seen some general guidelines although I am not really sure if they are completely true, for example: 1 - Linear classifiers for binary ...
2
votes
0answers
22 views

feature encoding for numerical features that depend on binary feature

I am working on a bounding box localisation problem where I need to detect whether a particular object is in an image, and if it is, I need to regress the coordinates of the bounding box. Currently, ...
0
votes
0answers
6 views

Clustering features of a class based upon the difference between features of a reference class and the particular class across multiple datasets?

I want to separate (generalized separation) the features of several classes based on the difference between the features (floating point values) of the particular class and a reference class across 7 ...
0
votes
0answers
10 views

Why one of the features is dominating all rest of the features in my trained SVM?

I have been given a task to train the SVM model on conll2003 dataset for Named Entity "Identification" (That is I have to tag all tokens in "Statue of Liberty" as named entities ...
1
vote
0answers
44 views

How to prepere dataset for binary classification (anomaly detection?) on timestamped sensor data (multiple files)?

my goal is to make prediction (good or bad data) on sensor data. I tried a lot, but failed to shape my data to get the desired output. scenario: I have multiple timestamped (time as it self is not ...
1
vote
0answers
26 views

How to use Lat/Lon in machine learning prediction?

I am research meteorologist working with hurricane model forecasts of track (millions of lat/lon pairs) and their verified lat/lon pairs (essentially where the hurricane actually went). With the ...
1
vote
1answer
18 views

How to incorporate static variables into ML

I have to establish an ML-based model where I predict precipitation in a complex terrain using multi-year daily observations from 50 stations. Besides a dozen of continuous variables, predictors ...
1
vote
1answer
20 views

How should I engineer features for Named Entity Identification task?

I was working on Named Entity Identification (not recognition) task. In this NLP task, given a sentence, model has to predict whether each word (aka token) is named entity or not. The dataset used ...
1
vote
0answers
37 views

Distance calculation for nonlinear features

Dear Data science community, Please see the attached. I plotted my data using t-SNE. In the figure, group A and B are 100% separable with random forest model. I want to calculate the distance of ...
0
votes
2answers
32 views

How to deal with Different Shapes of X_train and X_test after OneHotEncoding?

I am trying to perform OneHotEncoding as well as feature scaling on my training and testing data separately, steps I did: ...
1
vote
2answers
26 views

When Does Feature Selection Takes Place?

I have a dataset where there are categorical features as well as numeric features, and I have to perform OneHotEncoding, Normalization and feature selection on it. In what order should I perform these ...
0
votes
1answer
35 views

Best Way to find the important features for the model [duplicate]

I have data with 245 Features and almost all of the features are categorical. I would like to know what will be the best approach to find the important features for training the model. I know I can ...
0
votes
0answers
8 views

Best Practices For Dealing With This Scenario

I'm presently building a spam classifier. The model is unable to even overfit the training set at present. To investigate, I plotted the distributions of the model's features, and compared them across ...
1
vote
4answers
275 views

Creating & handling large matrices in python? [closed]

I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to ...
0
votes
0answers
15 views

What is the best way to create input data samples using in XGBoost for predicting number of next days that customer will come back to store

I'm building the tree-based model like a XGBoost to solve the problem about customer purchase cycle. And I think, I will build 2 models which one is predicting the customer will come back to store in ...
0
votes
0answers
5 views

Feature selection algorithm for psychometrics, when there is several predicted variables

I'm on a psychometric study. It is a survey. All variables are on a scale of 7. So these are considered as continuous variables. I have this dataset: 600 features 100 predicted variables 100 survey ...
0
votes
1answer
16 views

Method of choosing features for better clustering?

I'm working on a project where I need to cluster data. After doing all the usual steps (in no distinct order: one-hot/BaseN encoding categorical data, doing a Quantile Transform due to none of the ...
0
votes
1answer
33 views

How to feature engineering after getting test data in deployment?

I am kind of confuse about this topic of feature engineering. I am trying to make an web app in which people can upload test data as csv. Now I am confuse about how to do feature engineering after ...
0
votes
0answers
16 views

Can a rolling mean filter be used to denoise in data processing for ML algorithms?

if I have a data of the type ratio and I have taken care of the obvious errors(for eg string in the column has only integral values) can I use a rolling mean filter for it? Or perhaps binning then ...
0
votes
0answers
21 views

How to use time series forecasts as input features?

I have a time series dataset containing daily data like below. Let's assume that I would like to make some forecasts of my temporal serie (x) and use it as a second feature feature (f) to predict the ...
0
votes
0answers
85 views

Preserve column order after ColumTransformer

I'm using Pipeline and ColumnTransformer modules from sklearn library to perform feature ...
0
votes
1answer
45 views

Preprocessing , EDA , and Feature Engineering

What is the difference between EDA, Feature Engineering, and Preprocessing? The main purpose is to make the raw data suitable for modeling. In EDA, we are cleaning the data and so does the ...
3
votes
1answer
108 views

What is the point of generating new features (linear or non linear) out of existing features in a dataset?

During feature engineering, we can create new features out of existing ones by using arithmetic operations albeit linear or not. Let's say we have two features x and z. We can then create (engineer) a ...
1
vote
0answers
13 views

Rule of Thumb for number of observations required to train a model with n independent variables?

I am aware adding more features to a model leads to overfitting of a model. Is there a rule of thumb for minimum number of rows required to build a model with n features in order to build a ...
0
votes
0answers
8 views

Any reliable dimension reduction implementations available to address class overlapping scenario?

I am currently resolving a class overlapping problem in machine learning and while running some class separation experiments I have observed that Linear Discriminant Analysis (LDA) is able to perform ...
0
votes
1answer
24 views

How to pass variable length data as feature to a neural network?

I am working on building a model to classify the type of touch the user makes (Long Press, Left Swipe, Right swipe, and so on). I have data with features that characterize the user's touch, like ...
0
votes
0answers
21 views

Where can I find real world examples of feature construction for reinforcement learning?

Since I am relatively new to the field of reinforcement learning and I read through the classic book Reinforcement Learning: An Introduction by Sutton et. al., in Chapter 9 the authors mention several ...
0
votes
0answers
31 views

Data Transformation for Machine Learning Regression Task

I am performing a ML regression task, using XGBoost Regressor. I am using financial time series data, namely the Close price of the EUR/USD exchange rate which I will transform into geometric log ...
0
votes
0answers
30 views

A feature is still right-skewed after log scaling. How should it be normalized for machine learning?

I've attached two images below of a heavily right-skewed feature - call it x. I log scaled x, but it is still right-skewed and ...
0
votes
0answers
12 views

Removing categories with low sample size

I have a categorical column with 4 unique labels: Left ventricular hypertrophy Normal rest ecg Wave abnormality Out of 831 rows, only 4 of them include wave abnormality. It is a really low sample ...
0
votes
1answer
16 views

Aggregation of low level features for a classifier

The objective is to predict router fail/no fail (1/0) in a future time window with all the data collected over the last hour (i.e. binary target) The data is received at two different levels: Router ...
0
votes
0answers
4 views

Should I generalize categorical features if the algorithm handles over-fitting well?

I'm referring to Kaggle feature creation exercise . The data frame contains a column(MSSubClass) that contains these unique values: ...
0
votes
0answers
17 views

Feature Engineering on 3 dimensions data

I'm doing a task where I was given 3 features (a1, a2 and a3) and 3 heavily unbalanced classes. I tried many balancing techniques like SMOTE and undersampling. None of them gives me a reasonable ...
3
votes
1answer
104 views

Finding if an outcome is predictable

Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
0
votes
1answer
25 views

Would it be a good idea to use PCA output as input in models?

I have some dummy variables that indicate the occurrence of an event. There is so many of them, so I used PCA on them, and it appears some of them are rather correlated together. Would it be a good ...
0
votes
1answer
145 views

Is feature importance in XGBoost or in any other tree based method reliable?

This question is quite long, if you know how feature importance to tree based methods works i suggest you to skip to text below the image. Feature importance (FI) in tree based methods is given by ...
0
votes
0answers
11 views

How do you aggregate features of lists (pooling alternatives)?

Is it possible to reduce non-correlated multi-dimensional data over features to 1D data? A working option is pooling (mean/min/max) over an embedding vector (n samples of embeddings of m dimensions). ...
0
votes
0answers
18 views

Year/Month as a feature in Random Forest Classifier

I need to include a Maturity Date feature in my scikit-learn RandomForestClassifier model. Since the day is too specific, I'm thinking of having a number with the ...
0
votes
0answers
20 views

payment data prediction at test time

I have the payment data of the client. I want to predict the prob of customers paying late with target classes being 0-30 days, 30-60 days, 60-90 days, and 90+ days based on this paper. The features I ...
1
vote
1answer
16 views

Time Series Target variable taken at much lower sample rate than input features

I have a regression problem that involves predicting a patient's blood pressure from a range of vital sign readings including PTT, PPG, and HR. Each of these input features has been taken at the same ...
0
votes
1answer
37 views

Feature engineering: The more features I add the better RMSE I get?

I have a model with 7 features, I'm trying to figure out if I can improve the performance of this model by adding additional features. So I'm relying on the RMSE to measure the accuracy of my ...
1
vote
1answer
16 views

Traditional alternatives to Caputure Words Sequence information in NLP

What were the traditional/earlier methods in which NLP researchers captured the word sequence information through feature engineering? I know the current methods which rely on deep learning models ...
1
vote
0answers
29 views

Dataset format for Transformer text-generation

I'm trying to find some tutorials on training Transformer for generating comments on articles. So far, I found an article showing how to train GPT2 as a chat-bot. Input files in that example are given ...

1
2 3 4 5
11