I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with only around 11% being classed as "Raised". So far, the Random Forest classifier is providing the highest level of accuracy but the True Positive Rate/Recall is sitting at around 40% and I can't seem to increase upon this. I have been provided with a list of phrases that should they be contained in the email content, then in all likelihood a ticket needs raising. Looking for some tips as to the best method to create a new feature based on phrase matching? Has anyone any experience in the best methods for doing this?


The problem with imbalance is that the optimizer can get a very good score by declaring everything 'not raised'. You need to cheat with your training data by removing that incentive. I would suggest a training set that is balanced 50/50 between the classes. Your evaluation set can still be representative, which will give you a sense of how it'll generalize.

  • $\begingroup$ I’ve used SMOTE to create a 50/50 balance and have achieved 92% recall but this has been at the expense of accuracy at 79%! $\endgroup$ – Sql_Pete_Belfast Aug 21 '19 at 19:13
  • $\begingroup$ I'm actually working on a very similar problem ('read' JIRA tickets and determine if they impact my team) and went through something similar. I've been using SpaCy and Prodigy for the job, rather than sklearn. They have some neat tools in there, including one that gives you a sense of whether more training data would help. It works by seeing how much the precision and recall lift based on the number of examples you train with. That might be a profitable avenue to pursue. $\endgroup$ – oneextrafact Aug 22 '19 at 19:27
  • $\begingroup$ Also, it would be useful to investigate your false positives to figure out why the model might be misclassifying them. Once you have that sense, you can augment your training set by including them as 'not raised' examples that the model can learn from... $\endgroup$ – oneextrafact Aug 23 '19 at 12:27
  • $\begingroup$ I have an idea why the false positives are being created. Some duplicate emails were labeled once correctly and subsequent reminders were labeled incorrectly. I've tried to remove the duplicates where i can but i think i've missed a few. Do you have any links with examples for the spacy implementation? $\endgroup$ – Sql_Pete_Belfast Aug 23 '19 at 17:00
  • $\begingroup$ Just fyi, Prodi.gy is a paid tool. To me it was worth it because I wasn't having much luck training SpaCy on my own, but YMMV. Link to text classification examples: prodi.gy/features/text-classification $\endgroup$ – oneextrafact Aug 27 '19 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.