I am training a CNN model for a three-class classification problem. To do this, I'm gradually unfreezing more convolutional blocks of a pre-trained Resnet-18 network. The thing is that after unfreezing a block (let's say block 3, and preceding blocks), the performance on the validation set did improve but the performance on the test dataset did not improve (as opposed to block 2, and preceding blocks).
I'm now wondering whether or not it is 'justified' (so to speak) to keep unfreezing blocks to see how this affects performance on the test dataset. I feel like I'm just capitalising on some kind of luck that the model fits the test dataset better just by chance.
In short, I guess my question is whether I should choose my models on validation performance or test performance?