# Tag Info

Accepted

### Why does data science see class imbalance as a problem for supervised learning when statistics does not?

It's generally not related to Data Science but what goes around; typically all sort of bad practice relating to laziness / looking for short term rewards. I wouldn't say DS is pushing for it but ...
• 2,319

### Deal with overlapping classes in classification modeling

You don’t have the information to reliably classify. When you have more features, this is hard to assess, but with just the two, you can visualize in a scatter plot like you have to see that the ...
• 3,979

### Preserving / fixing class imbalance

There are many references and guidance for this kind of question. I recommend you consider the metric that matters most. Consider recall precision, and/or F1. You might consider creating class weights ...
• 131

### Solve tough clustering problem with overlapping clusters

The Problem is that many clustering algorithms focus on distances (between points, clusters and so on). Especially at the connection between the two desired clusters, distances between points are ...
• 1,574

### Why do we need hyperparameter tuning in Scikit learn? Doesn't sk learn models by default give best model?

You are wrong. RandomForestClassifier does not try any hyperparameters. You need to give it the specific value for each of its hyperparameters. Given such ...
• 27.1k

### Preserving / fixing class imbalance

Using Bayes theorem, we can write the posterior probability of class membership as: $P(C|x) = \frac{P(x|C)P(C)}{P(x)}$ The posterior probability of class membership is the ideal information we need ...
Accepted

### Xgboost model predicting extreme values for events and non-events | Overfitting

This is not necessarily overfitting, but it may indicate data leakage i.e you are passing information to the model that is not supposed to be there it may be: Information that is generated after the ...
• 3,009

### Car Make and Model detection

For detecting the make and model of cars from images with high accuracy across a large number of classes, I would recommend a convolutional neural network (CNN) architecture tailored for fine-grained ...
• 3,009

### ROC curve for a perfect model, why is AUC 1.0?

The difficulty with ROC curves is to understand what happens when the threshold varies. There is no summing, the curve only depends on how many instances have TP/FP/TN/FN status for every threshold. A ...
• 25.6k

### Why is my LSTM model not predicting well when predicting labels for a new dataset?

Your model is likely overfitting and you get an inflated validation score of 98% because of a data leak. Your idea of setting aside 3 of the time series for testing is good, but you missed to do the ...
• 4,243
Accepted

### Train/test split of data, stratified based on label, but ensuring no athletes are In both train/test sets

I think you may use the concept of groups as implemented in scikit-learn. In GroupShuffleSplit you may set a column of groups. Then the split won't happen across groups. Either a group as a whole is ...
• 66

### Fixing class imbalance vs Over-detecting in test data

"binary classifiers tend do better in terms of F1 scores when the class imbalance is at least reduced. However, this leads to over-predicting in the test data" This suggests that you need ...
Accepted

• 12k
Accepted

### Data augmentation technique not working correctly

You should measure the performance on the same dataset to be a good head-to-head comparison. I suggest you create a non-augmented test set first, then train a simple SDG from the train set, and then ...
• 146

### Public Email Classification Dataset but not Spam vs Ham

If I am understanding correctly, you want to create a model that takes an email body and assigns some probability to a pre-specified set of classes (feedback, complaint, lost and found, etc.) ...
1 vote

### Adding multi-image context to a CNN

You can perform individual image classification using as input the whole set of related images as well. In order to do this, your initial input will be some 3d array of stacked 256x256 matrices and ...
1 vote
Accepted

### How can I approach this transactions data problem?

There are so many things to take into consideration but my answer will focus on some divergent thoughts to help you with your modeling. 1 - I would start by understanding the underlying distribution ...
• 36
1 vote
Accepted

### Why do we need hyperparameter tuning in Scikit learn? Doesn't sk learn models by default give best model?

FYI, as of scikit-learn 1.3.2, the RandomForestClassifier's default hyperparameters are: n_estimators=100 criterion='gini' max_depth=None min_samples_leaf=1 ... ...
• 1,107
1 vote

### Xgboost model predicting extreme values for events and non-events | Overfitting

A high score on the test set does not indicate overfitting. See Why 100% accuracy on test data is not good? ; you're not quite reaching perfect performance, but you're quite close, and in that seeing ...
• 12k
1 vote

### How to build a categorization system without a target variable?

Welcome to Data Science! The first step is to make clear for yourself and future models the output you are looking for. It appears it's clear in your mind which tests to priorities but it's not in the ...
• 378

Only top scored, non community-wiki answers of a minimum length are eligible