We are a team of 3 participating in a university competition for a deep learning course. The competition involves a binary image classification task where we have to predict leaf diseases on a (5200, 96, 96, 3) shaped dataset.
We have tested various models and submitted them to the competition site, where they are evaluated on a hidden test set. Depending on the accuracy, precision, and recall obtained, we are placed on a global public leaderboard.
What I find strange is that, regardless of our efforts, the models we upload seem to perform in a completely different and apparently random way than what we would expect.
For example, we have fine-tuned 4 EfficientNet models (B4, B7, V2S, V2M, V2L), and they all achieved between 80% and 90% accuracy on the validation set (we didn't implement callbacks that could influence that, like LRScheduler). However, when uploaded to the leaderboard, they behaved unexpectedly. The worst model achieved the best score (83%), while the best and a few others dropped to (75% to 77%).
I've also noticed that when we change the train/validation split, we get somewhat consistent performance for the same identical model on our validation sets (there's not much variance; a good model stays good, and a bad model stays bad, from what we've observed). But when we upload it to the leaderboard, we again get significantly different scores (the same model went from 77% to 83% just by changing the seed of the splitting function). I know this is supposed to happen in general, but a 6% difference in accuracy seems too much, especially in a competition. In a scenario where two teams develop the same identical model, the team with the "best split" can beat the other, which shouldn't happen, I think.
We have tried balancing the dataset with oversampling, data augmentation, callbacks, fine-tuning, transfer learning, and so on, but the behavior persists.
The hidden test set should belong to the same dataset, so these differences seems quite strange to me.
It's almost impossible to do model selection locally because we can't predict how they will perform on the hidden test set. (I forgot to mention that we only have 6 submissions per day, which might seem like a lot, but if we have to upload the model every time we change the batch size or the augmentation strategy, it becomes a burden.)
We have already contacted the organizers, but obviously, they are not being very collaborative since it is a competition.
I also have to mention that in the dataset 196 completely unrelated random images were added and I proceeded to remove them since I assumed that in the real world (represented by the hidden test set, if I've understood well) those things should never appear.
Anyway, I'm just a student, so I'm here to ask you if this is expected behavior or not. In the second case, what can we do to mitigate this problem, and what might be causing it?
