0
$\begingroup$

I've been doing some experiments with setting different seeds on transformer training, and wanted to understand why I see so much variance.

Am I correct in thinking the random seed only influences two things when transformer training:

  • The random sorting of the training data;
  • Which connections get removed by dropout.

Oh, and the initial value of the randomly-initialized weights.

If so, that implies there is no stochastic element at all when using the model for inference?

Maybe my question would be better phrased as: what functions use randomness, so I can search a codebase for them, and confirm where they are used?

(I'm mainly looking at PyTorch implementations such as fairseq and huggingface; but I am assuming tensorflow implementations handle random numbers the same way.)

$\endgroup$
2
  • $\begingroup$ Please, consider marking the answer as correct or commenting on what makes you think it is not. $\endgroup$
    – noe
    Jan 8, 2023 at 9:09
  • $\begingroup$ @noe Thanks for the nudge. I had wanted to run a test of trying to confirm I get repeatable results if I drop dropout, and don't sort training data, but might be a while before I get to it. $\endgroup$ Jan 8, 2023 at 12:40

1 Answer 1

1
$\begingroup$

Your understanding is correct, both regarding the stochastic elements at training time (weight initialization, training data ordering, dropout) and that there are no inherent stochastic elements at inference time in the model itself.

Now, assuming we are talking about either a full transformer with encoder and decoder (e.g. for machine translation, like the original formulation) or a mere transformer decoder (e.g. a causal language model like GPT-3): note that the model itself does not define the decoding strategy (i.e. how to generate tokens), so you can choose to use a deterministic strategy (e.g. greedy decoding, beam search) or to use a stochastic decoding strategy (e.g. normal sampling from the output multinomial distribution, nucleus sampling, top-k sampling).

Also, there are some practical nuances to take into account, like some non-deterministic behaviors of specific implementations in CUDA (see this).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.