0
$\begingroup$

My code:

class spatial_dataset(Dataset):  
    def __init__(self, dic, root_dir, mode, transform=None):


    self.keys = dic.keys()
    self.values=dic.values()
    self.root_dir = root_dir
    self.mode =mode
    self.transform = transform

def __len__(self):
    return len(self.keys)

def load_ucf_image(self,video_name, index):
        path = self.root_dir + '/' + video_name.split('_')[0] + '/' + 'v_' + video_name + '/'

    print(path+'Image'+str(index)+'.jpg')
    img = Image.open(path+'Image'+str(index)+'.jpg')
    transformed_img = self.transform(img)
    img.close()

    return transformed_img

def __getitem__(self, idx):
    print(idx)
    if self.mode == 'train':
        video_name, nb_clips = list(self.keys)[idx].split(' ')
        print(video_name, nb_clips)
        num_clip = int(nb_clips)
        print(num_clip)
        clips = []
        clips.append(random.randint(1, int(num_clip/3)))
        clips.append(random.randint(int(num_clip/3), int(num_clip*2/3)))
        clips.append(random.randint(int(num_clip*2/10), num_clip+1))
        # for i in range(10):
        #     clips.append(random.randrange(1, num_clip, 1))
        print(clips)
        

    label = list(self.values)[idx]
    label = int(label)-1

    if self.mode=='train':
        data =[]
        for i in range(len(clips)):
            key = 'img'+str(i)
            index = clips[i]
            data.append(self.load_ucf_image(video_name, index))
        data = np.array(data)
        label = np.array(label)
        sample = (data, label)

    return sample

dic.training is dictionary, which have; {'videoname_1 framecount_1': label1,'videoname_2 framecount_2' : label2, .....}

training_set = spatial_dataset(dic=dic_training, root_dir=data_path, mode='train', transform = transforms.Compose([
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])
                ]))
        print('==> Training data :',len(training_set),'frames')
        print("training_set: ",training_set[1][0])

Until above, it is running but when I use this below code it shows me error (mentioned below)

train_loader = DataLoader(
            dataset=training_set,
            batch_size=BATCH_SIZE,
            shuffle=True,
            num_workers=num_workers)

    for z,i in train_loader:
        print(i)

The error it is showing me:

enter image description here

It is something because I am using dictionary to store values, I then converted it into the numpy arrays and list but still it is showing me this error I do not know where it is going wrong.

$\endgroup$
2
$\begingroup$

In your spatial_dataset class, dict.keys() is called to get the keys. This is known to cause pickling errors such as the one you are experiencing. The link above shows that you can handle this in three different ways:

  1. Iterate over the dictionary directly
  2. Use in for containment
  3. Convert to the type you want via iterator

If this does not work for you, you can use the following link, where in another Stack answer there is a brief discussion on tracing this pickling error using the "dill" python package.

I suspect that this might also happen to your call dic.values() call as well, so I would see about changing that if all else fails. Hope that helps.

$\endgroup$
3
  • 1
    $\begingroup$ I solved it by making num_workers = 0 $\endgroup$ Aug 8 at 16:43
  • $\begingroup$ I see. I'm new to the DataLoader function but it sounds like by setting num_workers to 0 that you removed any parallel processing. Is that correct? If that is so, then it may just be a suitable workaround if its not taking too long way. $\endgroup$ Aug 8 at 21:48
  • $\begingroup$ yes, I solved it. $\endgroup$ Aug 9 at 7:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.