0
$\begingroup$

At present, there is an app classification task, the input is the function description of the app, and the two labels are the major category to which the app belongs and the small categories under the major category. For example, the major category of FIFA online is sports, and the smaller category is football, which is included in sports. These two labels will affect each other, what machine learning model should be used to solve this supervised task?

The solution I thought of is to construct two layers of output. The output of the first layer is a large category, and the latter is a small category. The two layers are connected by a multi-layer neural network, but the details including the loss function and the training optimization method are still doubtful. Hope Get answers from everyone, thank you.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Your problem has 3 main sections as below:

  1. Text data (function description) as input.
  2. Looks like a multi-label & multi-class classification problem
  3. There is hierarchy/dependency between the two classifiers (Parent and sub category)

Based on this information, I would suggest you have a look at BERT/transformers based multi-label & multi-class classification work

This could be a helper blog: https://towardsdatascience.com/multi-label-multi-class-text-classification-with-bert-transformer-and-keras-c6355eccb63a. This will help you set up first two points from above.This itself could give you decent performance.

Now for the 3rd Part(dependency between 2 classifiers): I would think about concatenating output of first classifier(dense layer) with BERT's CLS output(768 dim vector) and using this as input for your second classifier.

For the loss function, just use cross-entropy for each classifier and add them for final loss. You can backpropagate from this loss (this allows you to think about doing a weighted average also if you like).

But during training, you might want to think a little bit about teacher forcing, if your model is not learning properly.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I tried searching for a pytorch based blog to help you with this, that does both multilabel and multiclass classification. But couldnt. But if you could find a way to do this with pytorch huggingface transformers, it would be good learning, as huggingface's transformers is a great library. github.com/abhimishra91/transformers-tutorials/blob/master/… fyr $\endgroup$
    – Narahari B M
    Commented Oct 8, 2021 at 1:18
  • $\begingroup$ Just be warned, the transformers blog mentioned in my first comment doesnt do both multi-label and multi-class. You would need to edit BERTClass to add 2 classifiers, then in forward function: concatenate bert cls embedding to classifier 1 output and feed that to classifier 2. Use cross entropy loss for each classifier and add them(or do weighted average) $\endgroup$
    – Narahari B M
    Commented Oct 8, 2021 at 1:22
  • $\begingroup$ Thanks to your three-point summary of the problem, I have got the collective name of this type of problem-hierarchical multi-label text classification, and inquired about related papers, and now share it with you. $\endgroup$
    – Paul Ji
    Commented Oct 8, 2021 at 11:30
  • 1
    $\begingroup$ Hierarchy-Aware Global Model for Hierarchical Text Classification $\endgroup$
    – Paul Ji
    Commented Oct 8, 2021 at 11:32
  • $\begingroup$ HFT-CNN: Learning Hierarchical Category Structure for Multi-label Short Text Categorization $\endgroup$
    – Paul Ji
    Commented Oct 8, 2021 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.