1
$\begingroup$

I have simulated a neural network with different learning rate, ranging from 0.00001 to 0.1, and recording each test and validation accuracy. The result i obtained is as below. There is 50 epoch for each learning rate, and i note down the validation accuracy at the last epoch, while the training accuracy is computed throughout the process.

Learning rate: 0.00001

Testing accuracy: 0.5850

Validation accuracy at final epoch: 0.5950


Learning rate: 0.0001

Testing accuracy:0.6550

Validation accuracy at final epoch: 0.6400


Learning rate: 0.001

Testing accuracy: 0.6350

Validation accuracy at final epoch: 0.6900


Learning rate: 0.01

Testing accuracy: 0.6650

Validation accuracy at final epoch: 0.6700


Learning rate: 0.1

Testing accuracy: 0.2500

Validation accuracy at final epoch: 0.2100

How does testing and validation accuracy influence which learning rate is better? Would a higher validation accuracy determine the most suitable learning rate for the model?

Hence, is it correct that 0.001 is the most suitable learning parameter since it has the highest validation accuracy at the last epoch?

$\endgroup$
1
$\begingroup$

You cannot select a parameter based on test accuracy, because the moment you do that, it becomes a validation accuracy as it has affected the final model. Therefore, you are always choosing based on validation accuracy.

As a result, the best result comes from learning rate 0.001, with the highest validation accuracy 0.6900. We have ignored Testing accuracy. If we select based on Testing accuracy, it becomes a validation accuracy.

Generally, a learning rate that is a looser at epoch 50, might be a winner at epoch 200. In other words, a slower convergence may lead to a higher accuracy. Therefore, this issue is worth considering too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.