I am building a model to predict orders, from its time series (univariate), for a company. I am working with 30 layers of 400 LSTM neurons each with the activation function hyperbolic tangent of Yann Lecun, 1 layer of 400 LSTM neurons with the relu activation function, and 3 layers of 500 neurons each, with a Relu Activation Function.

These values have had the best performance yet, but I don't know if there are typical values for training, or if it is an abnormally huge model.


Theoretically, a model should be big enough to have a low bias (avoid underfitting), but not too big as to have too high of a variance (avoid overfitting), called the bias-variance tradeoff

enter image description here

Whether the bias is too low for you or not depends on the true distribution you are trying to model. Practically, this means whether your model is getting a low training loss or not.

Whether the bias is too high or not depends on the true distribution and how much data you have. If your model achieves low training loss but high validation loss, then it's overfitting and thus has too high of a variance. In that case you need to either lower the complexity of the model or get more data.

If you ever reach a point where

1- reducing the complexity of the model will make training loss too high, and

2- increasing the complexity of the model will make validation loss increase

Then the only way to improve is to either change your model architechture/hyperparameters in some way or get more data.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.