1
$\begingroup$

I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000

Intuitively, it feels like using all the negative samples would prevent the classifier from learning invalid patterns (that might otherwise be learned using undersampling, for example).

Am I underestimating the effect of the imbalance on classifier performance?

$\endgroup$
  • 1
    $\begingroup$ Your data seems to be highly imbalance which will affect the model's performance significantly. The number of cases, where in the sample is positive but the model predicts it as negative, will increase. $\endgroup$ – Shubham Panchal May 31 '19 at 15:17
2
$\begingroup$

An imbalance of this magnitude is definitely a problem. Because machine learning algorithms penalize model complexity and try to maximize fit, a model that always predicts "False" will often outperform any other model because it predicts the right outcome for 99% of the data and is super simple!

What one should do in this case is downsample (http://www.simafore.com/blog/handling-unbalanced-data-machine-learning-models), train the model, and then scale the resulting predictions back up to reflect the imbalance in the original data.

$\endgroup$
1
$\begingroup$

Firstly any imbalance class text classification will have biased towards majority population and will result in overfit/underfit. You can use Smote, imbalancelearn library and imbalance from Scikit learn to optimize the low population and arrive at fair results, even if not good accuracy.

I hope this helps

$\endgroup$
1
$\begingroup$

I am also currently working on a binary classification problem with an imbalanced data set! Here's what I've found useful:

  • Use class weights. If you're using Keras you can pass this as an argument to model.fit(). Here is a notebook from Francois Chollet, creator of Keras, using this on an imbalanced data set for a binary classification problem.
  • Use a large batch size during training so that each batch will have at least a few True data points.
  • Use an appropriate metric. For example, accuracy is generally not a good one to use with imbalanced data - for example if you were trying to optimize accuracy, the model would ultimately learn to just classify a sample as False every time as it would end up with an accuracy of 99% which seems great but completely defeats the purpose of what you're trying to do. Area under the precision-recall curve is a good choice for imbalanced datasets, but I encourage you to do some reading and find a metric that works best for your problem.

Hope this helps and let me know if you have any more questions!

$\endgroup$
1
$\begingroup$

I was working on a similar problem and had found these two articles which cover everything that you need to know about handling unbalanced data. 1) https://towardsdatascience.com/practical-tips-for-class-imbalance-in-binary-classification-6ee29bcdb8a7 2) https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.