High loss and high accuracy on validation dataset at the early stage of training binary classifier

Question

I am training a ResNet50 network with simulation data and my validation dataset is the experimental data. The simulation data is not a 100% accurate representation of the experimental data. The purpose of this network is for binary classifier. I notice something very strange at the initial state of training as following:

The training cross-entropy loss is ~0.69, which is roughly equal to -log(50%) and the accuracy is ~50%. This logically makes sense, because the model basically hasn't learned anything and is just randomly guessing. The loss on the validation (experimental) dataset is also ~0.69, however, the accuracy is either very close to 0% or 100%. I understand this is partially caused by the difference between simulation (training) and experimental (validation) data, but it might be telling something deeper than that, for example, how the simulation dataset is different from experimental data. I couldn't figure it out, and I would love to hear any opinion. Please refer to the metrics below.

Answer 1

To me this is like heavy overfitting, certainly caused by the difference between training and validation data as you said. I also suspect that the test set might be too small, given the very strong the variations in accuracy. It looks like the result on the test set is almost random.

There's evidence that something is happening around epoch 20: clearly the model becomes able to classify the training data better, however it looks like this advantage doesn't apply to the validation data (it's not really clear).

Answer 2

Given the results that simulation data is not consistently transferring to experimental data, you could explore different optimization than gradient descent.

For example, genetic programming is another optimization technique. Genetic programming would take what is working (aka, parameters that perform 100% on the experiment data) and randomly modify them to explore the parameter space to find additional solutions.