1
$\begingroup$
Context

Working to deliver a POC on automated email classification (in customer service context) to tag emails as related to feedback, complain, lost and found etc. The tags are not entirely exclusive, but the goal of the model is to assign a weight to each of these tags for a specific email. Like based on the email body, it is 20% related to feedback, 70% complaint and 10% lost and found.

Now, ideally, I would start with my client company's real email inbox. But it is not a mature data company (for systematic consumption of their inbox data), and there are privacy issues yet to be resolved.

Question 1

Is there any publicly available email/feedback related dataset (with plain texts, and other optional features like timestamp etc.) that can be used to show some quick POC? Most email data I see are obviously spam-ham type, not in the domain I want.

Question 2

Any idea which model (best if pre-trained with well documented interface, like from HuggingFace) will be suitable for the task, with some scope for fine-tuning? I am personally more a software engineer with ML experience, not an NLP expert, but picking up as I go.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

If I am understanding correctly, you want to create a model that takes an email body and assigns some probability to a pre-specified set of classes (feedback, complaint, lost and found, etc.)

Regarding Q1, you should look on Kaggle, there's a chance someone might have done it. Otherwise, you will have to source your own data. This is a bit of a tough situation because the probabilistic weights that you are describing seem to be highly subjective (i.e. what does it mean for an email to be 70% related to feedback?). You will likely have to figure this definition out before you proceed.

Regarding Q2, you can go 2 routes:

  1. Assuming that you have the sourced data from Q1, use a discriminative transformer model architecture (like BERT) as your feature extractor, and for the final layer, use the softmax function. Just use the cross entropy loss as you normally would. You should get a vector, for each email, that describes the probability of something being a certain label. Treat this as a regression task and make sure that you retain the vector output (in a classification task, you would take argmax() of your vector, but not here).

  2. Use a generative text model (i.e. GPT, PaLM 2, Gemini, Llama) and prompt it to classify the email.

In all honesty, the second option is easier if you want a quick and dirty "proof of concept", but, in my experience, generative text models are notorious for hallucinating, even with the most conservative hyperparameters. If you need to incorporate this model prediction as a REST API within a system that would send these "weights" as requests to another service/API, then I highly suggest trying to synthesize your data, and doing method 1.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I can add an idea to the option 2. As using a generative model on the scale might be costly, you might use some Spam/Ham classifier first to dispose of Spam and use the model on the remaining one. $\endgroup$
    – Tomasz Witkowski
    Commented Apr 12 at 9:48
  • $\begingroup$ Option 1 will probably be cheaper. Easier to train and you don't need all the text generation part of the model if your goal is classification. $\endgroup$
    – picky_porpoise
    Commented Apr 12 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.