1
$\begingroup$

I am working on a multiclass classification which is to assign medical related queries of web search to certain departments of hospital.My classifier is based on the fastText.

I found for most conditions, the result is good enough say recall is 0.8 for Nephrology. However, for just one department, Dermatology, the recall is pretty low,like 0.5. Unfortunately, this label has most samples in the test data.

How can I improve the recall of one class while maintain the performance of other classes? Will ensembling method work?

$\endgroup$
  • $\begingroup$ Hello, welcome to the site! I would like to know what technique you have used for separating the data into test and train? $\endgroup$ – Toros91 Jan 15 '19 at 5:56
  • $\begingroup$ @Toros91 I just shuffle the original data and down-sampling the labels with too many samples and seperate train-test by 8:2 for each label. Acturally, I tried to get an balanced data by setting up a threshold for ~30000, any label with more than 30000 samples will be down sampled. However, in the original data, there are about 240000 samples with label Dermatology. I am not confident of my method to choose data, do you think it causes the problem? $\endgroup$ – leakey Jan 15 '19 at 6:12
  • 1
    $\begingroup$ so now when you do that I would suggest you to Stratified sampling which would help you in doing the sampling better and you wouldn't have problem while classification. Did you check if the levels in the test and train are same? $\endgroup$ – Toros91 Jan 15 '19 at 6:16
  • $\begingroup$ @Toros91 Thanks! I used disproportionate stratified sampling, maybe I should try to use proportionate stratified sampling for large classes. Appreciate your help. $\endgroup$ – leakey Jan 15 '19 at 6:26
  • $\begingroup$ One more suggestion is if you find a level less significant then it is not necessary for you to keep it in the training or testing set. You can use a new category called other when it is not classified into any class. can be segregated into that $\endgroup$ – Toros91 Jan 15 '19 at 6:28
0
$\begingroup$

Recall is the ability of a search model to find the correctly labeled items amongst all the items for a given query.

One common method to improve specific query results is to create a custom model. Create a model just for "Dermatology". That model can be tuned to increase recall without impacting other queries that can use the generic search model.

| improve this answer | |
$\endgroup$
0
$\begingroup$

A couple options:

  • Increase the confidence score corresponding to your class-of-interest until you reach the desired recall
  • Upsample the class you wish to have better recall on in the training set
  • Use class sensitive weighting ... make the loss associated to incorrectly classifying your class-of-interest higher than the others
  • Create two models, a binary model for your class-of-interest and a second model that predicts on everything but that class. Tune the "other" threshold of the binary classifier so that you have your desired recall. For all texts that have confidence scores less than the other threshold, classify them using your second model.
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.