X_train, y_train from ImageDataGenerator (Keras)

Question

Can I have X_train, y_train, X_test, y_test from data_generator? Here is my code:

data_generator = ImageDataGenerator(
    rescale = 1. / 255, 
    shear_range = 0.2, 
    zoom_range = 0.2, 
    horizontal_flip = True,
    vertical_flip = True,
    rotation_range = 180,
    width_shift_range = 0.2,
    height_shift_range = 0.2,
    validation_split = 0.2) 

train_generator = data_generator.flow_from_directory(
    train_data_dir, 
    target_size =(img_width, img_height), 
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset='training')

validation_generator = data_generator.flow_from_directory( 
    train_data_dir, 
    target_size =(img_width, img_height), 
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset='validation')

Answer 1

in python 2:

X_train, y_train = train_generator.next() X_test, y_test = validation_generator.next()

in python 3:

X_train, y_train = next(train_generator) X_test, y_test = next(validation_generator)

Answer 2

As per the above answer, the below code just gives 1 batch of data.

X_train, y_train = next(train_generator)
X_test, y_test = next(validation_generator)

To extract full data from the train_generator use below code -

step 1: Install tqdm

pip install tqdm

Step 2: Store the data in X_train, y_train variables by iterating over the batches

import tqdm
train_generator.reset()
X_train, y_train = next(train_generator)
for i in tqdm.tqdm(range(int(train_generator.n/batch_size)-1)): 
  img, label = next(train_generator)
  X_train = np.append(X_train, img, axis=0 )
  y_train = np.append(y_train, label, axis=0)
print(X_train.shape, y_train.shape)

Answer 3

Use

X_train, y_train = train_generator.next()
X_test, y_test = validation_generator.next()

y_train, y_test values will be based on the category folders you have in train_data_dir. Not values will be like 0,1,2,3... mapping to class names in Alphabetical Order.

Otherwise, use below code to get indices map

train_generator.class_indices
validation_generator.class_indices

Make sure they both are the same.

Answer 4

More of an indirect answer, but maybe helpful to some: Here is a script I use to sort test and train images into the respective (sub) folders to work with Keras and the data generator function (MS Windows).

import os
from glob import glob
from shutil import copyfile

############################
# Data stored at...
odir = "C:/origdir/"
# Target dir for test-train split
tdir = "C:/myimages/"
# Test-train-split (= x * maxsamples)
trainsize = 0.8
# Define max numer of samples for test-train
maxsamples = 2500

paths = glob(str(odir)+"*")

for p in paths:
    # get name of dir
    classname = p[p.rfind("\\")+1:]
    ###########################################
    # Gen dirs in tt
    # Check/create dir
    # TEST
    try:
        os.makedirs(str(tdir) + "/val/" + str(classname))
    except FileExistsError:
        pass
    # TRAIN
    try:
        os.makedirs(str(tdir) + "/train/" + str(classname))
    except FileExistsError:
        pass
    # ###########################################
    # COPY
    # train samples
    filelist = os.listdir(p)
    filelist = filelist[:maxsamples]
    tindex = int(trainsize*len(filelist))
    trainfiles = filelist[:tindex]
    testfiles = filelist[tindex:]
    # train
    for f in trainfiles:
        copyfile(p + "/"+  f, str(tdir) + "/train/" + str(classname) + "/" + f)
        #os.remove(p + "/"+  f)
    # test
    for f in testfiles:
        # get filename
        copyfile(p + "/"+  f, str(tdir) + "/val/" + str(classname) + "/" + f)
        #os.remove(p + "/"+  f)