Is __getitem()__ in a PyTorch Dataset restricted to always returning the same sample for the same index? I am thinking that the samples might be cached by some downstream tasks, for instance, so I am reluctant to do this, but is it actually not a problem?

(Context: this is for a Masked Language Modeling task, where I was thinking of having an epoch cover each sentence once, with random masks for each sentence. The next epoch would have different masks—so at the same index in the dataset.)


1 Answer 1


This is not a fully conclusive answer, but I understand that a PyTorch Dataset can return random samples for the same index. In fact:

  1. One natural use case is data (e.g. image) on-the-fly transformation. There is a (non-official) example on StackOverflow.

  2. Another argument in favor of random samples being allowed is that it is generally not a good idea to cache a dataset: this is the whole idea behind the Dataset and Dataloader classes, which allow the data to be fetched lazily (i.e. only when necessary).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.