Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Is __getitem()__ in a PyTorch Dataset restricted to always returning the same sample for the same index? I am thinking that the samples might be cached by some downstream tasks, for instance, so I am reluctant to do this, but is it actually not a problem?
(Context: this is for a Masked Language Modeling task, where I was thinking of having an epoch cover each sentence once, with random masks for each sentence. The next epoch would have different masks—so at the same index in the dataset.)
This is not a fully conclusive answer, but I understand that a PyTorch Dataset can return random samples for the same index. In fact:
One natural use case is data (e.g. image) on-the-fly transformation. There is a (non-official) example on StackOverflow.
Another argument in favor of random samples being allowed is that it is generally not a good idea to cache a dataset: this is the whole idea behind the Dataset and Dataloader classes, which allow the data to be fetched lazily (i.e. only when necessary).
