2
$\begingroup$

I will first tell you about the context then ask my questions.

The model detects hate speech and the training and testing datasets are imbalanced (NLP).

My questions:

  1. Is this considered a good model?
  2. Is the False negative really bad and it indicates that my model will predict a lot of ones to be zeros on new data?
  3. Is it common for AUC to be higher than the recall and precision when the data is imbalanced?
  4. Is the ROC-AUC misleading in this case because it depends on the True Negative and it is really big? (FPR depends on TN)
  5. For my use case, what is the best metric to use?
  6. I passed the probabilities to create ROC, is that the right way?

enter image description here

Edit: I did under-sampling and got the following results from the same model parameters: enter image description here

Does this show that the model is good? or can it be misleading too?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What are your class ratios? $\endgroup$
    – liakoyras
    May 27 at 6:22
  • $\begingroup$ @liakoyras 14k to 2k $\endgroup$
    – FjkgB
    May 27 at 6:37

2 Answers 2

Reset to default
3
$\begingroup$

The first model where the f1_score is around 61% can not be considered as a good model. You can achieve much better results than that. This can be seen in the second case (where you have downsampled the dataset), where the f1_score increases substantially.

Since your problem statement is to detect hate speech, you would have to decrease both, the FP and the FN or in other words, increase the precision and recall.

I would the say the metric in this case would be the f1_score which is a combination of precision and recall.

Also instead of downsampling, try oversampling. Or better yet, do neither and instead use other techniques to counteract the imbalance (think cross validation particulary RepeatedStratifiedCV, or maybe get more data for the minority class not by oversampling but from the authentic sources. )

Improve this answer
$\endgroup$
2
  • $\begingroup$ I will try to find more hate speech data, but I have a question: Is it a good practice to under sample? Is the sample still representative? Should I always try to make my data balanced even if it means to down sample? $\endgroup$
    – FjkgB
    May 27 at 6:40
  • 1
    $\begingroup$ When you oversample or downsample your data, the resulting sample will not be representative of the real dataset, since the real dataset is imbalanced and by downsampling your are balancing the dataset. Whether it is a good practice is a subjective matter. The best thing to do is try to get authentic data in increase the minority class as this would balance your dataset as well as not introduce any noise (which in the case of down or over sampling, you are basically introducing noise, albeit intelligent noise). $\endgroup$
    – spectre
    May 28 at 4:25
3
$\begingroup$

Most of your questions cannot be objectively answered.

Whether or not a model is good depends on what is the use for it.

Seeing how your classes are imbalanced, it definitely affects the metrics you presented. Do you care more about False Positives or False Negatives? What are the consequences of this? How many False Negatives are you willing to allow in order to have less False Positives?

Is it common for AUC to be higher than the recall and precision when the data is imbalanced?

This is an example of your model not being "as good" (given the caveats I mentioned). High ROC AUC means that your data can be ranked well while varying the threshold, which is to be expected since most of your data belongs in one class. But when considering precision-recall as individual metrics, at least one of those (precision if you have a lot of FP and recall if you have a lot of FN) will be more sensitive to the type of error you have, thus having lower values.

For my use case, what is the best metric to use?

F1 score is a pretty solid option whenever there are imbalanced classes, because as I mentioned it punishes both FP and FN.

But, by its definition, it is an average (harmonic mean) between precision and recall. If you care more about reducing a specific type of error, you can focus more on maximizing the more specific metrics (precision/recall).

I passed the probabilities to create ROC, is that the right way?

Yes it is. ROC is dependent on classification threshold, thus it needs to know the probability in order to be able to determine where to classify the sample given the specific threshold it checks each time.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the very good answer. Do the results I got after under-sampling mean that the model will perform good or is the sample not representative anymore? $\endgroup$
    – FjkgB
    May 27 at 7:24
  • 2
    $\begingroup$ I think it means that your model performs better on balanced datasets, which is to be expected. Training on imbalanced datasets has its benefites (for example reducing FPR) and in a lot of cases it is the only possible way due to lack of data, but it has the side effect that the model is heavily biased towards the majority class. For example if your model was always predicting Negative, in your 14:2 scenario it would be correct 6/7 times (86% accuracy), making it only slightly worse than the original model. That's why it is important to consider which is more important, FP or FN. $\endgroup$
    – liakoyras
    May 27 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.