2 votes

Enhance clustering with evaluation function

It seems to me that you are facing a metric learning problem. Here is a survey on the topic: https://people.bu.edu/bkulis/pubs/ftml_metric_learning.pdf In particular, the scikit-learn library for ...
gabalz's user avatar
  • 156
2 votes

Effect of feature selection when coupled with XGB models

Many aspects are at play here, mostly related to the data itself. Because with most feature selections, they are usually there to perform redundancy reduction. The redundancy is mostly defined with ...
timmy1691's user avatar
2 votes

Encode 10k features where each feature is having more than 500 categories

There is no single best technique for anything. You will have to try multiple techniques and see which one gives the best result. Also since your categorical variables are all high cardinal variables, ...
spectre's user avatar
  • 2,020
2 votes

Multiple Hypothesis Testing in feature selection process

In a basic statistical model like a logistic regression on the features, this kind of univariate feature selection is problematic. Briefly, a candidate feature can be fairly unrelated to the outcome ...
Dave's user avatar
  • 3,698
2 votes

Is there a standard data science workflow/decision tree?

David Robinson uploaded dozens of such analyses to youtube, as recordings of his screen while doing the analysis. Here is the link to the playlist. These come from "Tidy Tuesday" (see the ...
noe's user avatar
  • 25.7k
2 votes
Accepted

If my target variable is binary, is it better to use Pearson's or Spearman's for my correlation vector?

Pearson's correlation coefficient measures the linear relationship between two continuous variables and assumes that the data is normally distributed. Spearman's correlation coefficient measures the ...
Pluviophile's user avatar
  • 3,768
2 votes
Accepted

Different Algorithms for 50-50 A/B Testing

You can reformulate your previous even/odd split as bit testing of the binary representation of the customer ID: for the first feature, you took the bit at the first position (the least significant ...
noe's user avatar
  • 25.7k
1 vote

Different patterns in dataset

The short answer is that you can only tell after you've tried. I think you'd find that information reassuring, since you mentioned you're in the early stages of learning. To address your specific ...
idnavid's user avatar
  • 111
1 vote

Why the order of the fearures affects synapse LightGBM predictions?

the order of the features ... affect the predictions of the model LightGBM's model representation refers to features by their positional index. When passing in data to generate predictions, that data ...
James Lamb's user avatar
1 vote

PCA for complex-valued data

While I'm late in the game, I felt that an deeper insight to this question could be provided for those who would like to use the above mentioned algorithms (matthiaw91 and Alex) on the data. ...
R_N's user avatar
  • 11
1 vote
Accepted

Feature selection for propensity model

This is your problem : can you even have a column of training data be dependent on the label we're trying to predict? It might seems a bit weird but I would take the problem differently. The first ...
Ubikuity's user avatar
  • 626
1 vote
Accepted

Unclear points on projection type and selection of distance metric in feature extraction for a set of scenarios

A1 - in Scenario IV, the data clusters are not linearly separable in 3d, but presumably would be once projected onto the 2d manifold (the manifold assumption or manifold hypothesis). So learning the ...
brewmaster321's user avatar
1 vote

Approaches to dataset, whose elements have different size

Do you expect the size of the dataset to be highly correlated with the classification into clusters? If so, then it could be a straightforward approach to simply count the number of columns and use ...
brewmaster321's user avatar
1 vote

Feature selection / missing values

Random Forest: Random Forests are often used for feature selection in a data science workflow. This is because the tree-based strategies that random forests use, rank the features based on how well ...
Harshad Patil's user avatar
1 vote

Сan a subset of features perform better than the base set?

If by 'work better' you mean 'having less generalization error', then yes it is possible. Consider the extreme case that we have 100 features, but only 1 of them is truly causally relevant to the ...
lpounng's user avatar
  • 998
1 vote
Accepted

integration of Feature Selection in Pipeline

Pipeline 2 If you do preprocessing (selction, transformation, ...) inside the Pipeline, it will be part of the cross-validation and only trained on the training dataset. Doing so will prevent some ...
Broele's user avatar
  • 1,352
1 vote
Accepted

Feature Selection - determining the significance of imbalanced categorical data column

To answer your question: First do some analysis over it. Like chi-square test. Try to create a model like RandomForest model, so that you can draw the feature importance, and then you can see how ...
Harshad Patil's user avatar
1 vote

Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value

Here are few suggestions that you can determine on your dataset to drop the columns Drop columns which have same value throughout i.e. that contains a single value Determine variance for each column ...
Kriti's user avatar
  • 363

Only top scored, non community-wiki answers of a minimum length are eligible