I have a corpus of 23000 documents that need to be classified into 5 different categories. I do not have any labeled data available to me, just freeform text documents and labels(yes, one-word labels, not topics).

So I followed a 2-step approach:

  1. Synthetically generate labeled data (using a rule-based labeling approach, obviously the recall is very low, ~ 1/8 documents are labeled)
  2. Somehow, use this labeled data to identify labels for other documents.

I have attempted the following approaches for step 2:

  1. Topic modeling on data classified using rules to extract significant terms and using significant terms to label the remaining documents.
  2. Finding significant terms using sentence embedding
  3. Using sentence embedding as features for my classifier

But I haven't been successful in getting good results for my document classifier. Are there any other methods that can be used to classify the documents?

All help is greatly appreciated.

  • $\begingroup$ Is this data set corpus publicly available? I am looking for a corpus of text documents for my dissertation. I'll be really helpful if you can share some sources to find it. I have already check kaggle/google data sets and other famous online sites to find data sets. $\endgroup$ Jul 21, 2022 at 9:35

1 Answer 1

  1. You could simply encode the documents using BERT and cluster the documents based on their content provided they sufficiently different in terms of the kind of content they contain.

  2. Another approach would be to train a document segmentation model which would segment documents based on their semantic structures and then classify the documents based on their masked skeletons. This however would require a large dataset to train. Fortunately you can find one online called PubLayNet. Augment that with a few representations of your documents for better generalization over the test set.

I've read about the second approach being implemented to classify patents, legal documents, research papers etc. With good results. However it would take a long time to train.

I'd recommend simply clustering the documents based on their text embedding (point 1) and then naming the clusters. If that doesn't work satisfactorily, try the deep learning method for document semantic masking.

  • $\begingroup$ Thanks for your reply, @tehem. I have tried the first approach, albeit using GloVe vectors, but it didn't give very good results. (I basically averaged out the glove embeddings of each token in the document to get document embedding). The second approach doesn't apply in my case as all the documents have the same layout. $\endgroup$ Aug 26, 2020 at 3:16
  • $\begingroup$ Okay. Nice. Then I think we would need a bit more info on the kind of documents they are and how a human is able to tell them apart. Is it by reading and understanding the content or can a human tell those documents apart by just looking at them.? $\endgroup$
    – tehem
    Aug 26, 2020 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.