Choosing a suitable learning rate based on validation or testing accuracy?

Question

I have simulated a neural network with different learning rate, ranging from 0.00001 to 0.1, and recording each test and validation accuracy. The result i obtained is as below. There is 50 epoch for each learning rate, and i note down the validation accuracy at the last epoch, while the training accuracy is computed throughout the process.

Learning rate: 0.00001

Testing accuracy: 0.5850

Validation accuracy at final epoch: 0.5950

---

Learning rate: 0.0001

Testing accuracy:0.6550

Validation accuracy at final epoch: 0.6400

---

Learning rate: 0.001

Testing accuracy: 0.6350

Validation accuracy at final epoch: 0.6900

---

Learning rate: 0.01

Testing accuracy: 0.6650

Validation accuracy at final epoch: 0.6700

---

Learning rate: 0.1

Testing accuracy: 0.2500

Validation accuracy at final epoch: 0.2100

How does testing and validation accuracy influence which learning rate is better? Would a higher validation accuracy determine the most suitable learning rate for the model?

Hence, is it correct that 0.001 is the most suitable learning parameter since it has the highest validation accuracy at the last epoch?

Answer 1

You cannot select a parameter based on test accuracy, because the moment you do that, it becomes a validation accuracy as it has affected the final model. Therefore, you are always choosing based on validation accuracy.

As a result, the best result comes from learning rate 0.001, with the highest validation accuracy 0.6900. We have ignored Testing accuracy. If we select based on Testing accuracy, it becomes a validation accuracy.

Generally, a learning rate that is a looser at epoch 50, might be a winner at epoch 200. In other words, a slower convergence may lead to a higher accuracy. Therefore, this issue is worth considering too.