I am trying to make a binary text classification model by using the encoder part of the transformer and then using its output to feed into an LSTM network. However, I am not able to achieve good accuracy on both the training set (92%) and the validation set (72%). Is my approach correct? Please tell me a better way to design the model and improve accuracy.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Your model is overfitting. You should try standard methods people use to prevent overfitting:
- Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values).
- If you have many input classes, label smoothing can help.
- You can try a smaller model dimension.
If you use a pre-trained Transformer (such as BERT), you, of course, cannot change the model dimension. In that case, you can try to set a much smaller learning rate for fine-tuning BERT than you use for training the actual classifier.
-
$\begingroup$ I have used dropouts and this is a binary classifier. I used a CNN after transformer encoder but its result is not good as compared to using LSTM. I would like to know whether using attention before the LSTM is good or after it. And what is the difference between the two, why the results will be different if so? $\endgroup$ Apr 25, 2020 at 5:15
-
$\begingroup$ is it that in the case of overfitting only the validation accuracy won't be good? Can't it be due to bad model design? $\endgroup$ Apr 25, 2020 at 5:18
-
$\begingroup$ From where I see Transformers are an alternative to LSTM cause with LSTM the gradient vanishes with long sequences, basically cause the Than and Sigmoid that make the ports work, and with Transformers it doesn't, through spatial positional encoding and multi-head attention (self-attention). Thinking about encoding it would make more sense to try something like Glove, Word2vec, fastText, etc; but consider that is just my intuition. $\endgroup$– user58746Dec 25, 2020 at 21:06
-
$\begingroup$ The validation data in my case was important, the problem is that the validation was 90+ % wrong classified, but the model was learning (low loss and LRAP ok)... I guess it's two things that are leading to this: the data and the architecture/model chosen to transfer learning, mainly the data and second the architecture/model cause 1 º the sentences were from a specific domain and 2 º the sentences were not in English. $\endgroup$– user58746Dec 25, 2020 at 21:10
-
$\begingroup$ why "set a much smaller learning rate"? The model is converging on training data, just overfits it. Increasing dropout / adding regularization should do the work. Other than that +1. $\endgroup$ Apr 18, 2021 at 8:12