1
$\begingroup$

I'm building a model to classify email content, to decide whether the email should lead to a JIRA ticket being "Raised" or "Not Raised". The problem I am having is the data is highly imbalanced with only around 11% being classed as "Raised". So far, the Random Forest classifier is providing the highest level of accuracy but the True Positive Rate/Recall is sitting at around 40% and I can't seem to increase upon this. I have been provided with a list of phrases that should they be contained in the email content, then in all likelihood a ticket needs raising. Looking for some tips as to the best method to create a new feature based on phrase matching? Has anyone any experience in the best methods for doing this?

$\endgroup$
0
$\begingroup$

The problem with imbalance is that the optimizer can get a very good score by declaring everything 'not raised'. You need to cheat with your training data by removing that incentive. I would suggest a training set that is balanced 50/50 between the classes. Your evaluation set can still be representative, which will give you a sense of how it'll generalize.

$\endgroup$
  • $\begingroup$ I’ve used SMOTE to create a 50/50 balance and have achieved 92% recall but this has been at the expense of accuracy at 79%! $\endgroup$ – Sql_Pete_Belfast yesterday
  • $\begingroup$ I'm actually working on a very similar problem ('read' JIRA tickets and determine if they impact my team) and went through something similar. I've been using SpaCy and Prodigy for the job, rather than sklearn. They have some neat tools in there, including one that gives you a sense of whether more training data would help. It works by seeing how much the precision and recall lift based on the number of examples you train with. That might be a profitable avenue to pursue. $\endgroup$ – oneextrafact 10 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.