40 votes
Accepted

train_test_split() error: Found input variables with inconsistent numbers of samples

You are running into that error because your X and Y don't have the same length (which is what ...
  • 1,153
39 votes
Accepted

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Taken from this post:https://stats.stackexchange.com/a/245452/154812 The issue There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors ...
22 votes
Accepted

How many features to sample using Random Forests

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following: The number of randomly selected features can influence the generalization error in two ways: ...
  • 6,035
14 votes

Is stratified sampling necessary (random forest, Python)?

If the number of values belonging to each class are unbalanced, using stratified sampling is a good thing. You are basically asking the model to take the training and test set such that the class ...
  • 1,769
13 votes
Accepted

Cross validation for highly imbalanced data with undersampling

You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and ...
  • 672
11 votes
Accepted

With unbalanced class, do I have to use under sampling on my validation/testing datasets?

Great question... Here are some specific answers to your numbered questions: 1) You should cross validate on B not B`. Otherwise, you won't know how well your ...
  • 6,708
10 votes
Accepted

When should we consider a dataset as imbalanced?

I think subsampling (downsampling) is a popular method to control class imbalance at the base level, meaning it fixes the root of the problem. So for all of your examples, randomly selecting 1,000 of ...
8 votes
Accepted

Why do we need to handle data imbalance?

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority. Let say that 1% of the population have that rare disease. ...
  • 2,593
7 votes

Imbalanced dataset: how to deal with test data?

You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy. In overall accuracy you ...
7 votes
Accepted

Exploration vs exploitation tradeoff to find a price that maximizes revenue

Without making any underlying assumptions you will not get anywhere. That said, there are multi-arm bandit strategies that try to optimize the rewards, there is a ton of research on this field. It ...
7 votes
Accepted

How are samples selected from training data in Xgboost

In Gradient Boosting the simple tree is built for only a randomly selected sub-sample of the full data set (random without replacement). While on the other hand, Random Forest the samples for each ...
5 votes

With unbalanced class, do I have to use under sampling on my validation/testing datasets?

For 1) and 2), you want to ...
  • 3,037
5 votes
Accepted

What is a "good" sample size

This is a tough question to answer without more information. I'm going to assume that this is for model building, but without more detail it's hard to recommend something. However, there are some ...
5 votes
Accepted

SMOTE for regression

I think SMOGN will work for your problem. The method is described in a paper titled: "SMOGN: a Pre-processing Approach for Imbalanced Regression". You can find it on arXiv. There is also a python ...
  • 66
5 votes
Accepted

Over-sampling: is my model over-fitting?

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification ...
  • 7,618
4 votes
Accepted

Avoid iterations while calculating average model accuracy

Yes, you can do all this using the Caret (http://caret.r-forge.r-project.org/training.html) package in R. For example, ...
  • 923
4 votes

Sampling from a multivariate von Mises-Fisher distribution in Python

It looks like you can sample the von Mises-Fisher distribution with that R package. Have you thought about calling R from within Python using the rpy2 package? I haven't tried this for myself, but ...
4 votes

K-Fold Cross validation confusion?

The accuracy is different because there are k-classifiers made for each number of k-folds, and a new accuracy is found. You don't select a fold yourself. K-Fold cross-validation is used to test ...
4 votes
Accepted

Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

OOB samples are a very efficient way to obtain error estimates for random forests. From a computational perspective, OOB are definitely preferred over CV. Also, it holds that if the number of ...
  • 6,035
4 votes

Why do we need to handle data imbalance?

Short answer: you need to deal with class imbalance if/because it makes your model better (on unseen data). "Better" is something that you have to define yourself. It could be accuracy, it could be a ...
4 votes

Keras negative sampling with custom layer

Ok, I just flipped the arguments in the loss: model.compile(loss=lambda loss, y_true: loss, optimizer='Adam') should be ...
  • 191
4 votes
Accepted

How to do k-folds in python whilst splitting into 3 sets?

I don't think there's a builtin way to do it, but the two methods you've mentioned combine pretty nicely to do the job: ...
  • 10.2k
4 votes

SVM SMOTE fit_resample() function runs forever with no result

This is expected and is not related to SMOTE sampling. The computational complexity of non-linear SVM is on the order of $O(n^2)$ to $O(n^3)$ where $n$ is the number of samples. This means that if it ...
  • 8,667
4 votes
Accepted

Why did sampling boost the performance of my model?

As you mentioned in a comment, you are upsampling before splitting the test set, which leads to data leakage; your scores are not to be trusted. The problem is that a given positive sample may be ...
  • 10.2k
4 votes
Accepted

Is sampling a valid way to reduce complexity?

I would get a sufficiently large random/representative sample and cluster that. To see what is such a sample, you will have to get two such samples and cluster them to get cluster solutions c1 and c2. ...
  • 362
3 votes

Sample selection through clustering

Clustering will be much too expensive for your purpose (most are O(n^2), and the good ones like HAC may even be O(n^3) - you won't be able to run them on 300k instances). Also beware of the ...
3 votes

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Honestly there is no intuitive way to understand why NCE loss will work without deeply understanding its math. To understand the math, you should read the original paper. The reason why NCE loss will ...
  • 39
3 votes

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

[I've added this answer as I think others miss the main theoretical gist.] Firstly, NCE and Negative Sampling (NS) serve different purposes: NS is a generic trick used to train a classifier if you ...
  • 336
3 votes
Accepted

Poor performance shown on Rare event modeling

If you are willing to use the caret package in R and use random forests, you can use the method in the following blog post for downsampling with unbalanced datasets: http://appliedpredictivemodeling....
3 votes

When should we consider a dataset as imbalanced?

Data imbalance problem ?? In theory, it is only about numbers. Even if the difference is 1 sample it is data imbalance In practical, saying this is a data imbalance problem is controlled by three ...

Only top scored, non community-wiki answers of a minimum length are eligible