2
$\begingroup$

Whenever I train a neural network I only have it go through a few epochs ( 1 to 3). This is because I am training them on a bad CPU and it would take some time to have the neural network go though many epochs.

However, whenever my neural network performs poorly, rather than have it go through more epochs, I try to optimize the hyperparameters. This approach has generally been successful as my neural networks are pretty simple.

But is training a neural network in this manner a bad practice? Are there disadvantages to immediately going to optimize the hyperparameters rather than running the neural network for more epochs?

$\endgroup$
2
$\begingroup$

Count of epochs is also a hyper-parameter. However, if you meant to ask what to choose to work upon, whether increasing epochs or some other methods like feature engineering , then below is my answer.

Increasing number of epochs is often attributed as hammer stroke to train the model where you yourself often don't have to think much about the data and deepNet does it for you generally. However, it comes at a cost of computational complexity, risk of overfitting, etc.

Instead, it is always advisable to work as much on tasks like feature engineering, tuning different other hyper-parameters while training. In fact, a data scientist is supposed to work on these things preferably and not just blindingly utilising the black-box concept of deepNet.

However, even if you choose to increase the count of epochs, you may utilise the technique called early stopping. It basically says to stop the training at a certain epoch if the validation loss does not improve which you can use without much thinking about the impacts of increasing the count of epochs because if the validation loss does not improve, it will stop there.

| improve this answer | |
$\endgroup$
1
$\begingroup$

I would prefer tuning hyperparameters rather than running the network for many epochs as running neural network for high number of epochs leads to overfitting which should be avoided.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The number of epochs is part of the optimization problem. Hence reaching optimized results entails considering the number of epochs as well as other hyperparamters.

| improve this answer | |
$\endgroup$
1
$\begingroup$

You should tweak with all hyperparameters, including number of epochs.

To speed up your training, use high learning rate η initially and reduce it when the loss saturates. This is recommended in Keras as a technique and they even offer a function doing that. This will allow you to train your network in less epochs in general. Be careful with the initial learning rate being too high because it can drive the learning process unstable (fluctuating loss function instead of monotonically decreasing).

Make sure that you also use the dropout hyperparameter, which has been proven to be very successful in boosting the performance of NNs, particularly in generalizing to unseen samples.

If you apply both the above, you can have an efficient network that is trained in less epochs than it would require otherwise.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.