As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch. I was thinking of using an existing Word2Vec embedding and placing a stack of Transformer encoder layers on top of it, with a final Dense layer for the classification. From my limited understanding of the Attention paper, this seems like it should work, but my model ends up performing very poorly (predicting all 0). Before I start looking for errors in my code, is there a conceptual reason for this architecture to fail?

Edited to add: I am performing multilabel classification, where each label has two classes. The data I have is from legal contracts. Each sample is a paragraph, and the different labels correspond to the presence or absence of legal concepts. I have a Word2Vec model trained on legal text, and the embedding is not part of the model - I tokenize, normalize and vectorize as part of the generator.

  • $\begingroup$ Welcome to StackExchange. Could you elaborate more on your problem? What kind of classification are you trying to perform? What are the classes you want predict, and what data are you using? At the moment, we don't have enough information to answer. Thank you $\endgroup$
    – Leevo
    Commented Aug 28, 2019 at 15:00
  • $\begingroup$ Thank you, Leevo. I will add further information to the question. $\endgroup$
    – MrMulliner
    Commented Aug 28, 2019 at 15:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.