Questions tagged [feature-construction]
The feature-construction tag has no usage guidance.
80
questions
0
votes
0
answers
24
views
propper feature encoding
I working with the following data set
also here is it's detailed description of "packet_dat" column
I can't understand how I can encode packet_dat column into proper feature so my ...
- 101
0
votes
0
answers
15
views
How do I reconstruct irregularly spaced lat-lon data (in the form of a matrix) using spherical harmonic fitting?
Introductory Links to spherical harmonics give a mildly rigorous introduction to the concept; but very few links describe how to actually use them. I have latitude-longitude data arranged the ...
- 11
0
votes
0
answers
35
views
Effect of removing duplicate and identical entries on dimensionality reduction
I have huge data with thousands of observations and millions of features. I need to do clustering so I use PCA/t-SNE/UMAP for dimensionality reduction followed by K-Means.
Currently, I retain only ...
- 101
1
vote
2
answers
41
views
Tips for scraping crypto data in the right way
I am scraping data from crypto site and want to use neural network algorithm for predicting data.
the way i save data is like these:
and there is bunch of other features like open/high/low/close for ...
0
votes
0
answers
40
views
How to structure spatio-temporal data for LSTM model input
I have a dataset representing species distribution (binary presence/absence) and I would like to use a LSTM to predict future species distribution. I am having trouble working out how I should shape ...
- 27
0
votes
0
answers
36
views
python detect asymptote of a curve
I have a curve like this :
typical second order equation. I would like to auto detect the asymptote value (the red line). Do you know if it exist a python built in function to do that (like ...
- 195
0
votes
0
answers
11
views
finite difference vs difference quotient for temporal features
Let's say I have daily snapshots for some feature f, e.g., daily event counters. Now I want to add a second order feature g that tracks how f changes over time.
This can be done in several ways, but I'...
- 1
0
votes
0
answers
43
views
If possible in a regression model, is splitting up the target variable to more target variable (using multiple models) have any drawbacks?
As in the title, my main goal is to have one or more boosting regression model for a target variable(s). Let's call the main target d . I can split it up like <...
- 25
2
votes
0
answers
79
views
Feature selection and model performance
Featuretools provides an automated way to generate features from your data, by providing relationships within your data and applying their so-called deep feature synthesis. It generates features like ...
- 121
0
votes
1
answer
101
views
Feature creation ideas for propensity models?
I'm working on a propensity model, predicting whether customers would buy or not. While doing exploratory data analysis, I found that customers have a buying pattern. Most customers repeat the ...
- 233
0
votes
1
answer
22
views
How to represent a time duration feature for cases where time is still counting
I have a problem where I am trying to classify the outcome of costumer complaint cases. I have several features already such as type of item bought, reason for complaint etc...
I am trying to add a ...
- 3
1
vote
0
answers
22
views
Best way to represent a version feature based on percentiles
We're training a binary classifier in AutoML, and one of the features consist of browser versions.
Currently these versions are provided "normalized" to the model, according to the ...
1
vote
1
answer
165
views
Algorithms for casual feature selection for continuous Y
Currently I have been trying to find some good algorithms for feature selection. Using correlation or other non casual type of method will not be the right way to do a feature selection. I'm am ...
- 204
3
votes
1
answer
247
views
Finding if an outcome is predictable
Suppose we are asked to predict something given a set of features, how do we know if that target is actually predictable? That is, how do we know if there is actually some relation between the ...
- 75
0
votes
1
answer
778
views
How to model a 3D graph into a vector so that I can feed it into a classification algorithm?
I have a 3D graph like below:
Ref: google images
It has 2 angles as X and Y and the Z axis is amplitude value (Each 3D graph is representing a pixel). I want to model this into some useful data ...
- 111
1
vote
0
answers
13
views
Tsallis entropy - advice needed regarding obtaining probability distribution
As is always the way I stumbled across Tsallis entropy on SO whilst looking for something completely different. This soon lead me reading all sorts of interesting but terse academic papers.
I am ...
- 111
1
vote
0
answers
133
views
How to add more weight to certain features?
I have extracted features from two types of signals. Prior to merging them to create one feature vector, I have computed an importance score of every feature within that type of signal.
I would like ...
- 11
1
vote
0
answers
23
views
Regression with a feature which has its own depth
I'm relatively new to ML/Statistical Analysis, and I'm facing a dataset structured like this
...
- 11
2
votes
1
answer
104
views
How do I combine predictions from classifiers for two different problem?
I am working on a classification problem for predicting whether the shipment is going to be late or not.
I would say the classifier is mediocre at predicting the positive class at the moment. But the ...
- 21
1
vote
0
answers
60
views
Best Feature extraction for at the end retrieving audio
I work on a machine learning algo, which basically learns sequences in an audio .wav and generates the most “logical” sequences. The algorithm learns features, so I generate MFCCs from the audio file. ...
1
vote
0
answers
72
views
Data Lineage/Traceability in Pipelines
I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal.
Does anyone ...
- 11
1
vote
1
answer
56
views
How to model a supervised recommender system with varying data
Suppose there are 2000 movies and a company wants to recommend some movies (for example, at most 5 movies) to each visitor. The objective is to learn how to predict which movie will be selected if a ...
- 201
1
vote
2
answers
77
views
Problem with a feature (normal distribution + peak around 0)
I have a feature that shows a characteristic of the instances. That characteristic can be present or not. If present it shows an almost normal distribution of values (actually a bit skewed to the ...
- 21
1
vote
0
answers
58
views
Handling highly correlated features [closed]
I have a data set of transactions and want to build a fraud detection model (classifier). Only 3 variables are given that could be used as input features. The number of transactions during past 3, 6 ...
- 121
1
vote
0
answers
23
views
Is there a common relationship between data inputs and the number of attainable features?
Is there a known relationship between the amount of information gain that comes from new data added to a dataset?
for eg: If I have a plant watering system that tells me:
An integer of how wet the ...
- 13
1
vote
1
answer
43
views
Imputing features with NA values in classification task
I currently have a dataset where each observation is a person's traffic ticket history over districts.
For each column, which represents a district:
1 represents that a person has received 1+ traffic ...
- 281
2
votes
1
answer
1k
views
How to handle a feature vector that could be variable length?
I would like to train a machine learning model with several features as input as X[] and with one output as Y. For example Every sample has a Data frame like this: ...
- 21
1
vote
1
answer
38
views
Importance of features
It is common to say in ML feature selection that features that are irrelevant in isolation can be important in combination with other features. Is there a simple example (one or two features) to ...
- 21
0
votes
2
answers
42
views
How to group categorical columns into similar types?
(Forgive me if the question is ill put. I am a novice in data science. Please comment or edit so that the question can be improved)
I have a dataset where we have to predict the future sale of a shop....
0
votes
1
answer
27
views
Is it suitable to change a feature by itself to generate an another feature?
Hi all I would love to hear your answers on this. Lets say I have two variables, voltage and current, in my data set. I could add another feature by squaring current (so as to calculate power).
Is ...
0
votes
1
answer
32
views
Representing user information
I have a task of representing a users feature matrix , i have features like gender , age etc but I also have a multivalue feature called as "movies watched" which is essentially another table of ...
- 191
0
votes
1
answer
66
views
Feature importance and deriving rules using tree based classification models
I have a dataset where I have categorical and continuous values with targets 0/1 (binary classification task). Since I need to find patterns and relationships in the occurrence of the event or target, ...
- 101
1
vote
0
answers
40
views
Extracting Features for Graph transformation
Suppose I have a directed graph G (V,E) whose transformation is defined by a library of patterns. Each vertex is of particular type.
The library of patterns contain subgraphs (g1,g2,g3 etc)and it's ...
- 141
2
votes
2
answers
370
views
Categorical features preprocessing for clustering
Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1
answer
101
views
NLP Feature creation from phrase matching
I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with ...
- 113
1
vote
1
answer
101
views
Depending samples in ad ranking and click rate prediction
I am struggling with the following problem:
Suppose we fit a machine learning model to model advertisers click rates. I used a Logistic Regression approach using a one-hot/dummy encoding.
We have ...
- 13
0
votes
0
answers
641
views
One feature - several units
I have a dataframe where one of the features is the Mileage expressed in some cases in $\frac{km}{l}$, while in others is expressed in $\frac{km}{kg}$, according to the combustion type of the car (so ...
- 369
3
votes
2
answers
461
views
How to treat the undefined values which make sense?
I'm currently trying to create a few features to improve the performances of a model. One of those features that I would like to create corresponds to the difference in days between a customer's ...
- 203
1
vote
1
answer
49
views
Potential problems with expanding training set
The problem is a binary classification one. My dataset contains users with activity over multiple days, where they all start with class 0 and can become class 1 after a certain activity (which is not ...
4
votes
2
answers
10k
views
Combining Latitude/Longitude position into single feature
I have been playing with two dimensional machine learning using pandas (trying to do something like this), and I would like to combine Lat/Long into a single numerical feature -- ideally in a linear ...
- 151
2
votes
2
answers
109
views
Creating a metric based on some features
I want to create a new metric based on some features but dont know how to start. I basically want to create a "job satisfaction level" metric based on some features. The features could be work hours, ...
- 71
1
vote
2
answers
2k
views
What are features for state-action pairs in RL?
I read this answer: What are features in the context of reinforcement learning?
But it only describes features for the state only in the context of cartpole, ie. Cart Position, Cart Velocity, Pole ...
- 574
1
vote
1
answer
1k
views
How to put multiple features into RNN input vector
I am trying to code a recurrent neural network (LSTM) to create music in python and was considering using multiple features instead of just the note pitch as an input into the network. Initially I had ...
- 37
3
votes
3
answers
2k
views
how to evaluate feature quality for decision tree model
Most of the tutorials assume that the features are known before generating the model and give no way to select 'good' feature and to discard 'bad' ones.
The naive method is to test the model with new ...
- 197
2
votes
1
answer
40
views
I want to create an additional feature(column) based on some manipulation of values from existing features
Consider my data-frame to be like this ('x','y','z' are features):
I want to create a python function which will take an expression as a string (something like this: 'x+y-2z') and create a new ...
- 23
3
votes
1
answer
318
views
Label Encode with pre defined classes [closed]
I have trained a model (Random Forest) and now I would like to use it to predict certain data on a particular day. I have a categorical column where there are some values (say a,b,c,d,e) over a period....
- 51
3
votes
1
answer
82
views
Using historical label as a feature in my ML model?
I am working on a predictive model to predict change in the price of an asset (up, down, no change). The labeling is based on the derivative of the price and is exponentially smoothed with an alpha of ...
- 131
1
vote
1
answer
629
views
How to deal with Optional Input
I'm from the vision world and only worked with pixels from 0-255, ignoring any side effects. My current problem is different, in the way that I cannot rely on the input data.
What my problem is:
I ...
- 13
1
vote
2
answers
592
views
how to represent location-code as a feature in machine learning model?
I am trying to predict the damage to a buildings after earthquake on a dataset which contains "district number" as feature. I think the feature will have a significant importance in predicting the ...
- 135
3
votes
2
answers
8k
views
Too much inputs = overfitting?
First question : can I mix different sorts of inputs types for example, height and age (of course my inputs are normalized)? in general, can we mix different types of inputs in a neural network ?
...
- 31