Methods and principles of selecting a subset of attributes for use in further modelling
2
votes
0answers
34 views
Representing a community as a vector
My setup is this:
Suppose I have transactional data over a large period of time. The parties of each transaction are labled, and I use Louvain algorithm for detecting communities (and sub-communities)...
0
votes
0answers
9 views
Combining features for explainable binary classification, imbalanced dataset with minimal manual checks
I'm building a binary classifier which should detect between "fake" and "genuine" objects for a certain domain. I have designed a dozen of numerical features which are typically large for fake objects,...
1
vote
1answer
11 views
Using historical label as a feature in my ML model?
I am working on a predictive model to predict change in the price of an asset (up, down, no change). The labeling is based on the derivative of the price and is exponentially smoothed with an alpha of ...
2
votes
1answer
32 views
Optimising Expensive Functions
I'm trying some different techniques to optimise a Boosted Gradient Regressor by using an evolutionary programming technique to try and find the most efficient set of features. So far I've been having ...
0
votes
1answer
31 views
How to implement feature selection for categorical variables (especially with many categories)?
I've been trying to get some ideas of how I could treat categorical variables when doing feature selection. Mainly I've been running Random Forest feature importance on Python for which preprocessing ...
2
votes
1answer
21 views
Feature selection for time series prediction
I'm working on an LSTM-based stock market forecasting problem and trying to figure out a way to select input variables.
When calculating correlation between variables (e.g. Close price of Tesla vs ...
0
votes
0answers
12 views
Relief Algorithm misses relevant feature?
I have a generated a set of imbalanced data: 70 samples of class 1 and 1000 samples of class 2. The target variable and all predictors are boolean.
I've made two predictors that should be relevant to ...
-1
votes
0answers
18 views
Does feature occurrences affect feature importance
Let’s say we have a data frame that looks like this: Df:
A B
0 45 78
1 5 34
2 3 0
3 56 0
4 34 0
5 23 0
Does the fact that we have more values ...
1
vote
0answers
20 views
How to classify a dataset into 5 classes even though the performance is low?
I have a dataset of 5 classes with 8 features, say A, B, C, D and E. Now when I try to classify these into individual classes, I get accuracy, specificity and sensitivity of approx 50-60%, which is ...
7
votes
5answers
114 views
When to remove correlated variables
Can somebody please suggest what is the correct stage to remove correlated variables before feature engineering or after feature engineering ?
0
votes
1answer
18 views
Best practices for selecting categorical features
I'm trying to create a classifier that will predict whether someone will attend an interview or not. Each data point is for a single candidate and contains details such as the location of the ...
0
votes
0answers
8 views
How to understand when partial dependence plot and feature importance don't agree
I'm checking partial dependence plot and feature importance on my binary classification using gradient boosting.
The top feature based on the feature importance is a flat line at partial dependence ...
0
votes
1answer
15 views
the error occurred while selecting feature using recursive feature elimination in sklearn
I tried to rank the feature using recursive feature elimination in sklearn. However, I got this error when using RFE. here are the error and code information.
...
1
vote
2answers
18 views
how to represent location-code as a feature in machine learning model?
I am trying to predict the damage to a buildings after earthquake on a dataset which contains "district number" as feature. I think the feature will have a significant importance in predicting the ...
0
votes
2answers
17 views
Text representation using TFIDF .toarray() freezes my computer. Too many features to handle?
I'm new to Data Science, so hopefully this question makes sense.
I have a dataset with ~50,000 rows. It consists of one column that has item category in it and one column with item description in it....
1
vote
0answers
28 views
How to do feature engineering for email cleaning / text extraction?
I have a large batch of email data that I want to analyse. In order to do that, I need to first prepare the data, as the messages are quite often >80% noise. Generally speaking, my dataset's structure ...
0
votes
0answers
17 views
Samples that share same features but have different labels/output values
I have built a clustering model based on numerical data, specifically time series clustering. Let's say using sales quantities (over time) of different products. In other words identify different ...
2
votes
2answers
41 views
Testing independence of random variables in Python
Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{...
0
votes
1answer
16 views
automatic feature selection
I have a lot (thousands are possible) of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the ...
3
votes
0answers
24 views
Feature importance over a subset of instance space instead of an entire instance space
I'm really curious if anyone has faced this problem before, or is it even widely studied at all.
Imagine we have a feature that isn't important (based on many widely available and textbook feature ...
0
votes
2answers
30 views
What to do if my target variable is column of lists? [closed]
How I can transform my target variable(Y)?
As it is list, I cann`t use it for fitting model, because I must use integers for fitting.
0
votes
0answers
15 views
Activity Detection
I am attempting to classify every point in a sound file as either being active or inactive. I have a binary mask of my training data which represents when they are active or not. Not every sound file ...
1
vote
1answer
37 views
Feature selection
Is it possible that out of several attributes $p$, only one attribute could be selected by a model in the feature selection and training phase? Then basically we are fitting a line. Basically, I was ...
0
votes
0answers
117 views
Isolation Forest Feature Importance
As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at ...
0
votes
0answers
46 views
Selecting a Specific Number of Features via Sklearn's RFECV (Recursive Feature Elimination with Cross-validation)
I'm wondering if it is possible for Sklearn's RFECV to select a fixed number of the most important features. For example, working on a dataset with 617 features, I have been trying to use RFECV to see ...
2
votes
1answer
193 views
Number of features of the model must match the input. Model n_features is `N` and input n_features is `X`.
I am new to data science and trying get some results. I'm applying Decision Tree Classifier. When my train and test datasets' size are not equal I get an error `...
1
vote
0answers
17 views
When to perform feature selection, how, and how does data affect choosing the predictive model?
Some background:
I am attempting to predict attendance at this place using various features I have collected. I added to these features by extracting binary information from some of them. For example,...
0
votes
0answers
9 views
Is it bad to add bias to the model?
I have this data where I have to predict the probability of a rare event, and there is a feature that always precedes the event when it turns negative. However, the magnitude and variance of the ...
0
votes
0answers
24 views
Dropping one of the one-hot encoded columns for Gradient Boost Methods/Decision Trees?
If I have the categorical variable like favorite_color and it has unique values red, green, ...
1
vote
2answers
58 views
Random Forests with complementary features
In my dataset, I have 2 features that are not only correlated but that makes sense only in the presence of each other. For instance, one would be the number of times a task was attempted and the other ...
1
vote
1answer
21 views
Interpreting lasso logistic regression feature coefficients in multiclass problem
I have a dataset with a large number of text features, where the target variable has three classes.
I have encoded the features using tf-idf. This has resulted in a dataset with an extremely large ...
1
vote
1answer
76 views
Should I eliminate all ID columns and similar columns from training data? [closed]
This is a basic question so bear my ignorance. I feel like they contribute collectively in no way to the target. This is for performance and accuracy.
The target is polar (0,1).
2
votes
3answers
387 views
Using python and machine learning to extract information from an invoice? Inital dataset? [closed]
DISCLAIMER: I have absolutely no background with machine learning/data science, and am unfamiliar with the general lingo of data science, so please bear with me.
I'm trying to make a machine learning ...
0
votes
2answers
51 views
Number of rows vs. number of variables
I am new to the ML/DS field. I started a project where I need to predict the age of users. I'd like to build two models, one predicting the age group (I created 6 age groups), and the other one ...
1
vote
1answer
37 views
What's the best way to select features independent of the model being used?
I am using tensorflow's DNNRegressor to model a multivariate regression problem. I want to form an optimal feature subset from a mixed bag of categorical and continuous features. What would be the ...
0
votes
1answer
23 views
Fitting Lasso model repeatedly on same train dataset
I am using lasso for feature selection. I have selected lasso parameters from grid search. Now, I used the the best model which I got to fit on training data set. Thus, out of 70 variables I got 66. I ...
1
vote
1answer
29 views
Getting different chi square values than sklearn function
I have a dataset which looks like this-
I am trying to use chi2 as a feature selction algorithm on it.Here is the code
...
1
vote
1answer
31 views
Getting wrong ch2 values from sklearn chi2
I have a feature vector table which looks like this-
This a table with 156 columns or features.I want to apply feature selection algorithm to this before applyinh my classification model.
This is ...
0
votes
1answer
25 views
Chi-squared for continuous variables
I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into ...
0
votes
1answer
16 views
Devise a single metric to measure retention of listeners using the service on a daily basis for the last month
I have dummy data for a music streaming service, the data is as follows:-
Monthly uniques = the number of unique listeners on each platform for the entirety of a given month
All-time uniques = the ...
1
vote
2answers
125 views
Determine useful features for machine learning model
I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:
What quantitative metrics ...
3
votes
3answers
418 views
Is Pearson coefficient a good indicator of dependency between variables?
Once I have been asked how would I calculate correlation between two time series. Since I am new to data science I answered: "I would just calculate the Pearson correlation coefficient". That wasn't a ...
4
votes
1answer
54 views
Clustering by common elements in a list
Suppose I have these elements:
a = [1, 6, 3, 4, 10, 32, 2, 54]
b = [20, 5, 14, 25, 18, 1]
c = [54, 3, 6, 12, 41, 1, 9]
d = [3, 4, 1]
e = [19, 20, 25, 5]
Each ...
1
vote
2answers
31 views
Clustering with arrays / vectors as features?
I am trying to cluster a large set of documents of which I have a DOC2VEC representation. But I want to cluster them with more features, thus resulting in having both a vector (...
1
vote
3answers
128 views
Time series feature extraction from raw sensor data for classification?
I have a tabular raw data from sensors with associated label and i want to extract the time series features like mean,max,min and std from the data all the sensor data and form another table or export ...
3
votes
0answers
30 views
Determine the most important documents for supervised learning
I have somewhat of a general/high level question.
Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
1
vote
0answers
22 views
Feature Engineering for POS register events: Anomaly Detection
I am working on a dataset with only one variable: POS Journal Events.It has different values such as Items ordered, Order Placed, OrderItems,Discount applied, Promotion applied, order voided etc. I ...
1
vote
0answers
25 views
Rapidminer and decision tree weights
In Rapidminer, are the decision tree's weights a measure of the "importance" of attributes in the splitting procedure ?
If yes, why is useful to know these weights ?
Are there better methods to know ...
3
votes
2answers
47 views
Adding new variable to model
Let's say I already have a logistic regression model (or other) with N number of explanatory variables and is 70% accurate. Now if there are other variables available, how would I test if the new ...
1
vote
1answer
15 views
How can I use geo location for predictive modelling [closed]
While determining the house price I don't want to throw away geo location as feature. At least I want to see if there is correlation. Geo-location is more like categorical variable. Is there a way to ...
