0
$\begingroup$

I am working on developing an MTL model. There are two tasks in the model, task1, and task2.

And I have used a custom data generator using following code:

def get_data_generator(data, split ,batch_size=16):
    imagePath = ''
    df =''

    if split == 'train':
        imagePath = '../MTLData/train/'
        df = data[data.dir == 'train']
    elif split == 'test':
        imagePath = '../MTLData/test/'
        df = data[data.dir == 'test']
    elif split == 'vald':
        imagePath = '../MTLData/vald/'
        df = data[data.dir == 'vald']

    pfrID = len(data.PFRType.unique())
    ftID = len(data.FuelType.unique())
    images, pfrs,fts = [], [], []
    while True:
        for i in range(0,df.shape[0]):
            r = df.iloc[i]
            file, pfr, ft = r['Image'], r['PFRType'], r['FuelType']
            im = Image.open(imagePath+file)
            im = im.resize((224, 224))
            im = np.array(im) / 255.0
            images.append(im)
            pfrs.append(to_categorical(pfr, pfrID))
            fts.append(to_categorical(ft, ftID))
            if len(images) >= batch_size:
                yield np.array(images), [np.array(pfrs), np.array(fts)]
                images, pfrs, fts = [], [], []

And model is fitted using:

H = model3.fit_generator(generator=get_data_generator(labels,'train',batch_size),
                steps_per_epoch=STEP_SIZE_TRAIN+1,
                validation_data=get_data_generator(labels,'vald',batch_size),
                validation_steps=STEP_SIZE_VALID+1,
                epochs=EPOCHS

)

Now, I want to create a confusion matrix for task1 (pfr) and task2(ft) for whole dataset.

I am stuck here as I am not using flow from directory structure and the data is too big to store in memory. Can someone help?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.