Theoretically, a model should be big enough to have a low bias (avoid underfitting), but not too big as to have too high of a variance (avoid overfitting), called the bias-variance tradeoff
Whether the bias is too low for you or not depends on the true distribution you are trying to model. Practically, this means whether your model is getting a low training loss or not.
Whether the bias is too high or not depends on the true distribution and how much data you have. If your model achieves low training loss but high validation loss, then it's overfitting and thus has too high of a variance. In that case you need to either lower the complexity of the model or get more data.
If you ever reach a point where
1- reducing the complexity of the model will make training loss too high, and
2- increasing the complexity of the model will make validation loss increase
Then the only way to improve is to either change your model architechture/hyperparameters in some way or get more data.