Questions tagged [feature-engineering]
the process of using domain knowledge of the data to create features that improve machine learning algorithms
393
questions
-1
votes
1answer
12 views
How to use id's in binary classification problem
I would like to predict for a given user (on a website) if he/she logs out from the website within ten minutes.
In terms of data, I have a user ID and timestamp of the latest post on the website.
...
0
votes
1answer
6 views
How to match features in new records for NLP BOW
I have a dataset that has 100,000 records
data in this dataset are 2 columns
1- Text
2- Class
When I apply BOW of my model I get big list of features
That is fine, I managed to work with them
my ...
0
votes
2answers
48 views
Why is count encoding effective in improving accuracy?
Can someone please explain why/how Count encoding of categorical features improve accuracy in classification when compared to simply label encoding them ?
I found one explanation in kaggle " ...
-1
votes
2answers
31 views
feature selection in sklearn?
the RFECV takes the estimator and finds which features are important according to that estimator. But, it gets slow when we are dealing with data with many features. But, if we use trees they give us ...
1
vote
1answer
20 views
How to use unigram and bigram as an feature on SVM or logistic regression [closed]
How to use unigram and bigram as an feature to build an Natural Language Inference model on SVM or logistic regression?on my dataset i have premise, hypotesis and label column. I'm planning to use the ...
0
votes
0answers
9 views
How to use the position factor in known data as a feature in recommendation surfacing?
The problem is recommending stories on a website, just below each story based on how similar the stories are and some historic data based on what recommended stories were clicked or not clicked. So ...
0
votes
0answers
10 views
Are there any techniques to explore the inter - target correlations and the input - output correlations in the Multi Traget Regression Problem
I am working on the Multi target regression problem in which there are 4 target variables and 17 input features.
I tried to use hard parameter sharing method to predict the target variables.
I am ...
0
votes
1answer
20 views
Does anybody know where this rule of thumb came from? Rule of them: embedding vector dimension should be the 4th root of the number of categories
I was taking an online ML course and the lecturer said that a rule of thumb for choosing the number of dimensions when embedding categorical data is the following
...
2
votes
1answer
15 views
When is it appropriate to split a dataset on a categorical value and generate $n$ models instead?
When doing regression or classification when faced with a categorical attribute with $n$ possible values there are two options:
Feed this attribute directly into your model.
Partition your data into $...
1
vote
1answer
17 views
Is there any problem with dropping only part of the OneHot generated features?
The one hot encoder adds more columns to the data, one for each category in the encoded feature. In the example below, the column City was transformed into 4 other ...
0
votes
0answers
30 views
How do I “tie” the weights of parallel layers of an ANN?
My problem is fairly simple, but I am not sure if my implementation in Keras does what I think it does. I have a set of features in a certain space of dimension N. I want to use an ANN to learn the "...
1
vote
2answers
27 views
Input with variable length Classification problem
I have a dataset with patient information with discrete labels (labels are stages of a particular disease) which needs to be predicted (Basically a classification problem).
The dataset looks ...
0
votes
1answer
16 views
How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?
I want to apply pd.cut as a transformer in a pipeline, like this:
...
0
votes
1answer
16 views
Logistic Regression Multi-level Independent variables
im trying to study logistic regression, when i did the target variable with all features,
i had the summary showing the p-values as usual, but one for the features has 60 level, another feature has 13 ...
0
votes
1answer
17 views
Which supervised machine learning algorithms assume normally distributed feature variables?
I want to understand the assumptions made by supervised machine learning models.
I've heard it said many times that 'you need to make sure your feature variables are normally distributed for your ML ...
0
votes
0answers
10 views
Using the input record's distribution as a composite input feature
I have a very interesting dataset that I need to use for doing regression. It is production data from stainless steel production and I have about 290 input features, so I need to start reducing the ...
1
vote
2answers
35 views
How to work with Log-transformation?
I'm beginning my data science journey and I've faced a challenge that confuses me a bit.
I have a set with few features and a target variable whose raw distribution is highly skewed.
I've read that ...
1
vote
1answer
21 views
Handling a Pandas Data Frame containing multiple non-ordinal categorical features
I'm currently trying to analyse a dataset containing multiple non-ordinal categorical features and a binary target variable. The table looks something like this:
...
1
vote
1answer
28 views
Normalizing dependent feature by one of the independent ones
I have a data set with three different features (x1, x2, x3) and I am going to use a regression model to predict y based on the features.
x3 is the total amount of money that a customer invest and y ...
1
vote
3answers
58 views
Feature importance in neural networks
Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, ...
0
votes
0answers
17 views
Defining the Target Value
im new to this community and it always helped me with my concerns, i looked for an answer but didnt find a clear one yet
im working on study for insurance default, the data i received is already ...
3
votes
1answer
18 views
How to perform data scaling/standardization on dataset containing grouped values?
So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example:
...
0
votes
1answer
26 views
What is the reasonable max number of features for LSTM?
I've been trying to find information on how many features I can use in LSTM.
I found that LSTM can handle up to 500-1000 timesteps, but what about features (sequences)?
Can I use 1000 features per ...
1
vote
1answer
15 views
How to feed key-value features (aggregated data) to LSTM?
I have the following time-series aggregated input for an LSTM-based model:
...
4
votes
2answers
149 views
Mean estimation for nested location data
I want to estimate the average income for a location. I have nested data in the following way: A block is inside a neighborhood, which is inside a zipcode, which is inside a district, which is inside ...
0
votes
1answer
16 views
How to pre-process the name String of a customer?
I implement logistic regression to predict if a customer is a business or a non-business customer with the help of TensorFlow in Python. I have several feature candidates like name, street, zip, ...
0
votes
1answer
21 views
How to group categorical columns into similar types?
(Forgive me if the question is ill put. I am a novice in data science. Please comment or edit so that the question can be improved)
I have a dataset where we have to predict the future sale of a shop....
1
vote
1answer
9 views
How and When features are attached to target label
I am using Mallet CRF library and having training set sequences like below.
...
2
votes
1answer
153 views
Tensor Flow Time Series Tutorial Question
In the tutorial, they normalize the data and say "The mean and standard deviation should only be computed using the training data"
What does this refer to? Why should you only use the training data?
0
votes
1answer
21 views
Should we do Feature selection in parallel with feature engineering?
I'm working with LightGBM on a large data set about 3M row and about 8 columns. When i ...
1
vote
0answers
22 views
Multi-dimensional Time Series Features
I am new to applying ML to time series data but I do have experience doing general supervised learning. I have time series that is multidimensional (so several variables over time) with one output ...
1
vote
0answers
15 views
Scaling monotonically increasing features between 0 and 1
To keep the test set blind to the neural network algorithm it is generally better to build a scaler based on the training set and then scale the test set on this scaler.
I am building an LSTM for ...
0
votes
0answers
10 views
Merging/Consolidation of duplicate records
I have large number of groups of duplicate records, which i need to merge to get a single master record per group.
Input dataset contains columns like - FirstName, LastName, FullName, AddrLine1, City,...
0
votes
0answers
13 views
Some doubts in sklearn.preprocessing.Normalizer ? Please explain in lucid manner without jagron
In sklearn.preprocessing.Normalizer it mentioned that “”Scaling inputs to unit norms is a common operation for text classification or clustering for instance. For instance the dot product of two l2-...
1
vote
1answer
28 views
Using ARIMA parameters when transforming time series to Supervised Learning
When forecasting time series one can change the problem from a classical time series (ARIMA type of models) to supervised learning (by adding lag features).
When the time series is long and you ...
0
votes
1answer
12 views
Fixed target values but more ID for clafssification
I have a data set where there the target column as values between 1-15. Now I need to predict that based on the features. the feature values are have unqiue IDs. but the target column is the same ...
1
vote
1answer
25 views
Discretisation Using Decision Trees
I'm new to the machine learning and working on a supervised classification problem. I used discretization process to transform continuous variables into discrete variables. So I followed this ...
0
votes
0answers
11 views
How to model predict on Tensorflow model that has feature columns?
So far all the coding examples using feature columns do not have examples of how to format their model.predict(...). I tried using raw string and also putting them ...
1
vote
1answer
20 views
Filling missing values for Embedded List in Python3
I searched for a similar question but I didn't come across. And I'm new in this area, I hope I explained my question well enough.
I have a dataset consist of text data. I store them in a list and ...
0
votes
1answer
34 views
Table transformation using Numpy, Pandas
I want to transform a table from this :
To This :
Explanation:
basically I'm trying to create a "Sankey chart" for that I need this type of format.
So From Table 1(1st Image): Using Date and ...
0
votes
1answer
33 views
Implementing sklearn's FeatureHasher on Unseen Data
For a little bit of background I have been working on a binary classification of health insurance claims and am implementing sklearn's FeatureHasher to vectorize categorical features, many of which ...
0
votes
1answer
23 views
Change format of table
How can I change the table format from this:
To This?:
Is there any way to do this in Python using pandas or NumPy?
0
votes
0answers
11 views
Best Practices for Boosting, Trees, Random Forests re. Number of classes in a feature and number of samples per class
I'm doing a regression to predict house prices using multiple algorithms and the h2o platform.
In particular, I'm using a variety of GBM, DRF, and GLMs.
I have a large amount of data from several ...
0
votes
0answers
24 views
Feature Selection for highly correlated feature
I have used a feature which has a high correlation with the target of 0.8 , but the accuracy of the model decreases in validation set when I add this feature. What might be the reason for decrease in ...
0
votes
1answer
17 views
Is it suitable to change a feature by itself to generate an another feature?
Hi all I would love to hear your answers on this. Lets say I have two variables, voltage and current, in my data set. I could add another feature by squaring current (so as to calculate power).
Is ...
1
vote
1answer
35 views
Understanding sklearn FeatureHasher
Wanting to understand "the hashing trick" I've written the following test code:
...
0
votes
1answer
21 views
Representing user information
I have a task of representing a users feature matrix , i have features like gender , age etc but I also have a multivalue feature called as "movies watched" which is essentially another table of ...
0
votes
1answer
28 views
What is “lag” in time series forecasting?
I'm studying machine learning (e.i. time series analysis). I encountered an Azure tutorial, Retail Forecasting.
https://gallery.azure.ai/Experiment/Retail-Forecasting-Step-2-of-6-train-time-series-...
3
votes
5answers
229 views
How can I handle a column with list data?
I have a dataset which I processed and created six features:
['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList',
'totalProducts']
And the ...
2
votes
0answers
13 views
Adding high p-value and low R square features in linear regression model to improve result
I am working on a linear regression problem. The features for my analysis have been selected using p-values and domain knowledge. After selecting these features, the performance of $R^2$ and the $...
