I have 100,000+ PDF healthcare documents from which I have extracted text. I would like to cluster these documents by type (e.g. pathology report, doctor visit notes, prescription orders, etc.)

The format and structure of the documents are unknown ahead of time. However, we can assume that the name of the document type occurs explicitly somewhere within the first 50 words of each document. E.g. A pathology report will contain the words "pathology report" somewhere near the top.

The contents of the documents are otherwise irrelevant. I simply want to cluster documents together by document type.

I do not know the names or number of document types ahead of time. I would like the algorithm to automatically determine the best number of clusters.

I'm not very experienced in machine learning, but I am a competent programmer. What's the best way to attack this problem? Thanks.

  • 2
    $\begingroup$ Read about "document embeddings" and "topic models". These are two ways of finding representations you can use to cluster the documents. If the structure of the document is also important, augment the textual features with visual features (e.g., number of columns, presence and relative size of a title... whatever you can think of). $\endgroup$ – Emre Jan 9 '18 at 18:24
  • $\begingroup$ @Emre I understand what document embeddings are, but that seems more relevant to the semantic content of the documents. I’m hoping to determine document types without having to delve into semantics. $\endgroup$ – brianberns Jan 9 '18 at 18:41
  • $\begingroup$ If the semantic content is not important go with visual features. If it is somewhat important go with semi-supervised learning or classification; use labeled data to teach the algorithm how important the semantic information is. $\endgroup$ – Emre Jan 9 '18 at 18:53
  • $\begingroup$ I'm hoping that the raw header text alone (first 50 words) is sufficient. Perhaps using TF-IDF as input to some sort of standard clustering algorithm? $\endgroup$ – brianberns Jan 9 '18 at 19:16
  • $\begingroup$ That's a semantic representation. Modern document embedding methods are a refinement of tf-idf. Try it and see. The problem is that your notion of clusters might not exactly align with the ones emerging from your representation. That's the problem labels solve. $\endgroup$ – Emre Jan 9 '18 at 19:32

"pathology report, doctor visit notes, prescription orders" these are classes, not clusters. Clustering may as well find "people with headache", "patients on Thursdays", ... you do not get to control this unsupervised.

So I'd rather suggest to do a classification.

For example, put all documents that contain "patient report" in the first lines into a separate set. Inspect the remainder for another keyword. Add another rule. If the data is as explained, 10 such rules may get the leftover documents down to a manageable scale, and they could eventually just be treated as 'others'.

There is nothing wrong with inspecting documents and adding rules if that solves your problem. You probably don't need to make this work on different document collections (say, law texts) ever.

| improve this answer | |
  • $\begingroup$ So there’s no way to do this via unsupervised learning? $\endgroup$ – brianberns Jan 11 '18 at 6:17
  • $\begingroup$ No, it does not know what you are looking for. It will likely cluster by something else. See: there probably is also “Thursday" somewhere in many documents. Why is the clustering by weekdays worse? $\endgroup$ – Has QUIT--Anony-Mousse Jan 11 '18 at 6:23
  • $\begingroup$ You can use unsupervised techniques such as frequent sequences to find candidate rules though. But there will be many non-relevant such patterns, too. It may be faster to just inspect remaining documents manually if just need to get the job done. $\endgroup$ – Has QUIT--Anony-Mousse Jan 11 '18 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.