As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch. I was thinking of using an existing Word2Vec embedding and placing a stack of Transformer encoder layers on top of it, with a final Dense layer for the classification. From my limited understanding of the Attention paper, this seems like it should work, but my model ends up performing very poorly (predicting all 0). Before I start looking for errors in my code, is there a conceptual reason for this architecture to fail?
Edited to add: I am performing multilabel classification, where each label has two classes. The data I have is from legal contracts. Each sample is a paragraph, and the different labels correspond to the presence or absence of legal concepts. I have a Word2Vec model trained on legal text, and the embedding is not part of the model - I tokenize, normalize and vectorize as part of the generator.