I am working with a Kaggle dataset with short Twitter messages as text input. I made a copy here. When testing LSTMS, GRUs, bi-directional versions of the GRUs, and the Encoder layers of a Transformer model, the best models were shallow and had few units. The average text sequence length was 14-15 according to my code. Is it a good assumption that smaller models or models with fewer units usually do better when processing short text?

There are problems with my guess.

  1. The correlation between layer counts and unit counts vs. the length of meaningful words or tokens is not constant. For bi-directional LSTMs, the best models had one layer and 8 or 16 units. Other unit counts from 64 to 1 did not perform as well. Using the transformer encoder,1 layer with 2 heads and 4 inner units performed roughly as well as 6 heads and 8 inner dense units. For 1-D Convolutions, 32 and 64-filter convolutions did the best. I tested common multiples of 2 and some in-between values if the upper and lower limits performed well, e.g. 24 if 16 and 32 filters did well.

  2. I did not sample a large range. I assumed that higher numbers of layers or units would make the model worse after reaching 128 filters for Conv-1D layers and 32 units for the LSTMs. This is because I did not have the computing power to test larger networks, not just that more units made performance worse. I also didn't give time to testing in-between values.

  3. The performances of my models might depend on the data set, but they could also depend on the model architectures I set up, i.e. what layers came before or after. How do I tell if a trend might be generalizable to other text classification data sets vs. something specific to my model architecture?

To summarize, I want to know how or what correlation there might be between model architectures or hyperparameters and the characteristics of the data set, like the sequence length and number of examples. If I test the same architectures on different data sets, would that help validate my guesses?

If you want, you can check the Notebook I linked. Some of the models I tested have been turned into comments.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.