I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels.
I have more than one sentence that needs to be considered by the model, so I need to find the best method to merge them into one sequence and keep it categorical so the fine-tuned model, BERT, RoBERTa, XLNet, etc; can consider the attention or positional relation between the tokens through the NN.
A resume of the Transformer idea:
Input embedding -> positional embedding -> multi-headed attention (self-attention) -> feed-forward -> normalization layer | output embedding -> positional encoding -> multi-headed attention (look-ahead mask) -> normalization layer -> feed-forward -> linear && probability distribution; besides its components encoder and decoder that uses some of the items described before
I have considered the obvious approach of merge the N featured into one string, but it doesn't give me a good result in terms of F1, I thought about polynomial features but I need to keep it as categorical and make relations between the features and them inverse it to categorical looks a lot of work for a lot of redundancy and word embedding, but these methods create a feature space where the categorical features have a meaning by its positions, which are normally discrete atomic symbols. Any tips on how to apply one of these methods with a different approach or a method that I don't know?