1
$\begingroup$

I have created the following class and function to create a 'custom' dataset. The use case was to sample every nth image from a data set to reduce the size of the data set while still maintaining representation from all the classes.

I tried subsetting and other similar techniques but these wouldn't have maintained the integrity across the various classes and the lost the index when implemented into a DataLoader. It works but it is neither neat nor tidy and takes some time to implement.

If anyone has any advice on how to tidy this up and consequently speed it up I would be forever grateful.

# Create a class to instantiate a dataset for use in a dataloader
class Cust_Ds(Dataset):
    def __init__(self, features, target):
        super().__init__()
        self.features = features
        self.target = target

    def __len__(self):
        return len(self.features)

    def __getitem__(self, idx):
        X = self.features[idx]
        y = self.target[idx]

        return X, y


def custom_length_df(data_set, every_nth_entry):
  red_ls_x = []
  red_ls_y = []
  p = range(0,len(data_set), every_nth_entry)
  for i in p:
    X, Y = data_set[i]
    x = np.array(X)
    y = np.array(Y)
    Xt = torch.Tensor(x)
    Yt = torch.Tensor(y)
    red_ls_x.append(Xt)
    red_ls_y.append(Yt)
  dataset = Cust_Ds(red_ls_x, red_ls_y)

  return dataset

Implementation is as follows:

trg_dataset = custom_length_df(trg_101, 10)
val_dataset = custom_length_df(trg_101, 37)
print(len(trg_dataset))
print(len(val_dataset))
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.