When deciding on the number of units and layers for text processing or time-series prediction I rely heavily on trial and error. First, I look for a reference or paper on the topic such as the white paper on transformers: Attention Is All You Need. Once I read why the standard is so-and-so, I code the standard. After that, I incrementally adjust the unit counts and number of layers one at a time. I am making wild guesses at that point. Maybe the number of meaningful or non-zero tokens could approximate the units required. If my sequence length is capped at 120 tokens, I'll see how long it takes to train a 128-unit LSTM, GRU, or Transformer model. I arbitrarily set the unit count to the lowest exponent value of 2 greater than the maximum number of tokens and then steadily reduce it. After that, I start adding layers. If the model has bad metrics, I keep adding layers. If the model takes more than a minute to train per layer, I reduce the number of units and layers. I tolerate long training times based on how much free time I think I have. Is there any way to search more systematically? My criteria are all arbitrary. I hope there is a way to calculate the layer and unit counts based on the input data or meta-data. I am trying this out with TensorFlow.
1 Answer
From my testing on 100,000+ models throughout a few hundred different problems, I have found the following:
Automate your testing across model types and architecture. Also, ensure you have something to track your runs and compare values. (MLFlow, WANDB, etc.)
Adding layers will not always help; sometimes, increasing layer weights, regularization, and recurrence have far more impact. There are numerous other techniques, including custom activation functions, for example, that can yield considerable performance improvements.
I have seen single-layer, seemingly simplistic models, tuned appropriately, perform as well as complex multi-head attention stacks. It all depends on the problem.
Datasets and scaling also significantly contribute to your results. Take time to identify the best approach to your problem. Try different scalers, try other features, see what performs consistently on test data, and measure the difference between the train/validation and test datasets.
Use statistical plots to view the deviation and see what metrics inform your model accuracy for the problem at hand.
There is no one-size-fits-all approach, at least not yet. Trial and error is the way. Automate the tests, refine, and tune the results. Once you have enough data from runs, you will understand what works well for each input sequence and correlation for the problem you are trying to solve.
-
$\begingroup$ Is there a way to automatically search through a range of layer counts and unit counts? I want to tune those parameters too. I want to minimize model run time. I can't think of a way to do that automatically based on the documentation of MLFlow and WandB. $\endgroup$ Commented Mar 5 at 6:21
-
$\begingroup$ I write code to do this for each problem, using a standardized approach, usually testing a few different architectures around baselines that have worked for me in the past. You can take a Bayesian approach using MLFlow/WandB. If you want an example, open a question for this specifically, and I will show you an example approach. $\endgroup$ Commented Mar 5 at 17:17
-
$\begingroup$ Thank you. This is the related question: datascience.stackexchange.com/questions/128194/… $\endgroup$ Commented Mar 7 at 1:47