44
votes
Accepted
train_test_split() error: Found input variables with inconsistent numbers of samples
You are running into that error because your X and Y don't have the same length (which is what ...
- 1,193
42
votes
Accepted
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
Taken from this post:https://stats.stackexchange.com/a/245452/154812
The issue
There are some issues with learning the word vectors using a "standard" neural network. In this way, the word ...
- 544
23
votes
Accepted
How many features to sample using Random Forests
I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following:
The number of randomly selected features can influence the generalization error in two ways: ...
- 6,422
14
votes
Is stratified sampling necessary (random forest, Python)?
If the number of values belonging to each class are unbalanced, using stratified sampling is a good thing. You are basically asking the model to take the training and test set such that the class ...
- 1,799
13
votes
Accepted
Cross validation for highly imbalanced data with undersampling
You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and ...
- 692
8
votes
Accepted
Why do we need to handle data imbalance?
You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority.
Let say that 1% of the population have that rare disease. ...
- 2,663
8
votes
Accepted
How are samples selected from training data in Xgboost
In Gradient Boosting the simple tree is built for only a randomly selected sub-sample of the full data set (random without replacement).
While on the other hand, Random Forest the samples for each ...
- 6,312
7
votes
Imbalanced dataset: how to deal with test data?
You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy.
In overall accuracy you ...
- 2,019
7
votes
Accepted
Exploration vs exploitation tradeoff to find a price that maximizes revenue
Without making any underlying assumptions you will not get anywhere. That said, there are multi-arm bandit strategies that try to optimize the rewards, there is a ton of research on this field. It ...
- 9,388
7
votes
Accepted
SMOTE for regression
I think SMOGN will work for your problem. The method is described in a paper titled: "SMOGN: a Pre-processing Approach for Imbalanced Regression". You can find it on arXiv. There is also a python ...
- 86
5
votes
Why do we need to handle data imbalance?
Short answer:
you need to deal with class imbalance if/because it makes your model better (on unseen data). "Better" is something that you have to define yourself. It could be accuracy, it could be a ...
- 6,096
5
votes
Accepted
How to do k-folds in python whilst splitting into 3 sets?
I don't think there's a builtin way to do it, but the two methods you've mentioned combine pretty nicely to do the job:
...
- 12k
5
votes
Accepted
Over-sampling: is my model over-fitting?
In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification ...
- 8,018
4
votes
Strategy for dealing with giant sample size
My link in the comment has useful advice. I'd like to emphasis:
This is a very well known fact in statistics
Large sample size is good. There's nothing wrong with more and better quality data.
It's ...
- 3,570
4
votes
Accepted
Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?
OOB samples are a very efficient way to obtain error estimates for random forests. From a computational perspective, OOB are definitely preferred over CV.
Also, it holds that if the number of ...
- 6,422
4
votes
train_test_split() error: Found input variables with inconsistent numbers of samples
Isn't train_test_split expecting both X and Y to be a list of same length? Your X has length of 6 and Y has length of 29. May be ...
- 286
4
votes
K-Fold Cross validation confusion?
The accuracy is different because there are k-classifiers made for each number of k-folds, and a new accuracy is found.
You don't select a fold yourself. K-Fold cross-validation is used to test ...
- 141
4
votes
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
[I've added this answer as I think others miss the main theoretical gist.]
Firstly, NCE and Negative Sampling (NS) serve different purposes:
NS is a generic trick used to train a classifier if you ...
- 416
4
votes
Keras negative sampling with custom layer
Ok, I just flipped the arguments in the loss:
model.compile(loss=lambda loss, y_true: loss, optimizer='Adam')
should be
...
- 191
4
votes
Oversampling before Cross-Validation, is it a problem?
I suggest having a read of this article. The article explains:
When upsampling before cross validation, you will be picking the most oversampled model, because the oversampling is allowing data to ...
- 447
4
votes
SVM SMOTE fit_resample() function runs forever with no result
This is expected and is not related to SMOTE sampling.
The computational complexity of non-linear SVM is on the order of $O(n^2)$ to $O(n^3)$ where $n$ is the number of samples. This means that if it ...
- 9,382
4
votes
Accepted
Why did sampling boost the performance of my model?
As you mentioned in a comment, you are upsampling before splitting the test set, which leads to data leakage; your scores are not to be trusted. The problem is that a given positive sample may be ...
- 12k
4
votes
Accepted
Is sampling a valid way to reduce complexity?
I would get a sufficiently large random/representative sample and cluster that.
To see what is such a sample, you will have to get two such samples and cluster them to get cluster solutions c1 and c2. ...
- 362
3
votes
Sample selection through clustering
Clustering will be much too expensive for your purpose (most are O(n^2), and the good ones like HAC may even be O(n^3) - you won't be able to run them on 300k instances). Also beware of the ...
- 8,054
3
votes
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
Honestly there is no intuitive way to understand why NCE loss will work without deeply understanding its math. To understand the math, you should read the original paper.
The reason why NCE loss will ...
- 49
3
votes
Accepted
Gumbel Softmax vs Vanilla Softmax for GAN training
Passing directly the output of the softmax is also common (among the few textual GANs out there), e.g. see the improved Wasserstein GANs (WGAN-GP).
With hard Gumbel-softmax (+ straight-through ...
- 27.2k
3
votes
Downsampling and class ratios
Downsampling means you sample from the majority class (the 98.5%) to reduce the imbalance between majority and minority class. If you keep the ratio constant you simply reduce your number of trainings ...
- 6,422
3
votes
Accepted
In Machine Learning, what is the point of using stratified sampling in selecting test set data?
When the distribution of your data is balanced or you have enough samples of each class, a normal shuffle split will work well. But if your data distribution is unbalanced and one of the classes is in ...
- 2,275
3
votes
Accepted
Sub-sampling so that sample statistics match population statistics
The term you are looking for is stratified sampling : https://en.wikipedia.org/wiki/Stratified_sampling. It's a way to sample from population that can be partitioned into sub-populations. More ...
- 2,348
3
votes
Accepted
Using SMOTE for Synthetic Data generation to improve performance on unbalanced data
First of all, you have to split your data set into train/test splits before doing any over/under sampling. If you do any strategy based on your approaches, and then split data you will bias your model ...
- 810
Only top scored, non community-wiki answers of a minimum length are eligible