1
$\begingroup$

I am solving a specific segmentation task, using two versions of U-net architecture - the first one being classic U-net and the other Attention U-net. Currently, I am trying to determine which one performs better for my specific use case.

The problem I am encountering is that because of the stochastic nature of model learning, no two results of the same architecture are alike. For example, let's say that I train two models (one after the other, using the same architecture, let's say U-net) and for the chosen testing data I get 98.5% accuracy for the first model and for the other I get 97.5% accuracy. Then I train another two models using the other architecture (Attention U-net) and for example get results: 97.6% and 98%.

The problem is that the third trained model could give me a value of perhaps 95%.

Because of the range of these "random" results, I don't know how to evaluate the architecture's performance and how to find the better one of the two.

What is the best approach for determining the performance of these two architectures and comparing them?

I am using keras.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Not sure what language/framework you are using for your experiments but, you should try to deep seed your models so that performance is decoupled from random as much as possible. See an example here: https://pytorch.org/docs/stable/notes/randomness.html

Ideally, you would want to have some sort of optimisation methodology in place, where attention: {True, False} is a hyperparameter alongside others.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I am using keras and so far I have done some research that non-deterministic process can't be achieved when using GPU (correct me if I'm wrong) $\endgroup$
    – A. Cimet
    Commented Nov 2, 2021 at 14:58
  • $\begingroup$ That is generally true, but you if you control for all other things you may be able to detect a difference that is significant enough to construct a hypothesis around. $\endgroup$
    – hH1sG0n3
    Commented Nov 2, 2021 at 15:20
  • 1
    $\begingroup$ Note that initialiazitation might be of great inflence im model performance. $\endgroup$
    – Pedro Henrique Monforte
    Commented Nov 2, 2021 at 15:26
  • $\begingroup$ Another option is to perform multiple tests until you get an acceptable margin of error to compare both models $\endgroup$
    – Pedro Henrique Monforte
    Commented Nov 2, 2021 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.