I am solving a specific segmentation task, using two versions of U-net architecture - the first one being classic U-net and the other Attention U-net. Currently, I am trying to determine which one performs better for my specific use case.
The problem I am encountering is that because of the stochastic nature of model learning, no two results of the same architecture are alike. For example, let's say that I train two models (one after the other, using the same architecture, let's say U-net) and for the chosen testing data I get 98.5% accuracy for the first model and for the other I get 97.5% accuracy. Then I train another two models using the other architecture (Attention U-net) and for example get results: 97.6% and 98%.
The problem is that the third trained model could give me a value of perhaps 95%.
Because of the range of these "random" results, I don't know how to evaluate the architecture's performance and how to find the better one of the two.
What is the best approach for determining the performance of these two architectures and comparing them?
I am using keras.