I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels.

I have more than one sentence that needs to be considered by the model, so I need to find the best method to merge them into one sequence and keep it categorical so the fine-tuned model, BERT, RoBERTa, XLNet, etc; can consider the attention or positional relation between the tokens through the NN.

A resume of the Transformer idea:

Input embedding -> positional embedding -> multi-headed attention (self-attention) -> feed-forward -> normalization layer | output embedding -> positional encoding -> multi-headed attention (look-ahead mask) -> normalization layer -> feed-forward -> linear && probability distribution; besides its components encoder and decoder that uses some of the items described before

I have considered the obvious approach of merge the N featured into one string, but it doesn't give me a good result in terms of F1, I thought about polynomial features but I need to keep it as categorical and make relations between the features and them inverse it to categorical looks a lot of work for a lot of redundancy and word embedding, but these methods create a feature space where the categorical features have a meaning by its positions, which are normally discrete atomic symbols. Any tips on how to apply one of these methods with a different approach or a method that I don't know?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy