I am somewhat new to text classification and I have some questions if you folks can help:

I have some text I need to be able to classify as belonging to a single class or not (usually 1-10 sentences long each). For the examples of the class, I have around 500 examples, but the non-class case can really be any text at all of which I have hundreds of documents with tens of thousands of sentences (which are not the class). What I have to do is be able to classify each of the sentences in each document as belonging to the class or not. The vast majority won't belong.

  1. I'm using a BERT based Binary Classifier (simpletransformer) to identify the text similar to (or exactly) the 500 class examples, does this seem reasonable/possible?

  2. How should I deal with the class imbalance of 500 to 10000's? I tried oversampling the minority class (my target), but it seems to overfit when I do that.

  3. What is the usual way of handling this particular use case? The 1-class anomaly detection doesn't seem to fit here, from what I can tell. Is there a similar NLP style training that works for this case? Or something else?

Would it make sense to just do a semantic similarity comparison of some sort? That is, just take the class examples, and for each sentence in a document, test to see how similar it is to each class example and if the text is "close enough" to any of the class examples, then it's a "hit"? this would seem slow... Is there a standard/good library for semantic comparison?


1 Answer 1


Well, I think that you identified the options and problems quite well.

  • The main problem in this kind of text classification task is that it's impossible to obtain a representative sample of the negative class.
  • Binary classification is certainly a reasonable option, but since a classifier learns to separate the two classes there's always a risk that some future negative example won't look like any of the training examples and end up misclassified.
  • One-class classification is also a reasonable option. By definition it's supposed to handle the open classification problem better, but it's not always the case in practice.
  • Calculating a similarity measure against the reference documents is possible, but it's not efficient so rarely convenient. The performance also depends a lot on the data and measure chosen, and of course there is the problem of determining an optimal threshold.
  • The class imbalance should probably not be treated by resampling, it's not going to solve anything. Imbalance is a problem only because the model doesn't find good indications in the features, so imho the only good option is to investigate and understand why the features don't help the model in cases of errors. Very often it's because the minority class is simply not representative enough.
  • $\begingroup$ Thanks for the response. I'll check into the One-class option and see if that might work, or test it. Do you think that using a BERT based classifier is a pretty reasonable method that I'm trying? $\endgroup$
    – superqd
    Dec 11, 2021 at 5:05
  • $\begingroup$ @superqd it's certainly a reasonable option but I always recommend starting with a traditional model like decision tree or SVM. It's simpler and it gives a baseline performance so one can check if the perf improves with a more advanced version like BERT. $\endgroup$
    – Erwan
    Dec 11, 2021 at 12:02
  • $\begingroup$ Thanks for the info! $\endgroup$
    – superqd
    Dec 12, 2021 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.