I would like to use a custom generator so that I can implement custom augmentations on my dataset in Keras. However, I built a generic generator (without augmentation) and am confused why it is performing way worse than the built-in fit function.

My generator looks like this:

def data_generator_plain(features, labels, batch_size):

    batch_features = np.zeros((batch_size, IMG_SIZE, IMG_SIZE, NUM_CHANNELS), dtype=np.float64)
    batch_labels = np.zeros((batch_size, NUM_KEYPOINTS * 2), dtype=np.float64)

    while True:
        steps = len(batch_features) // batch_size
        for i in range(steps):
            for j in range(batch_size):
                batch_features[j] = features[(i*batch_size)+j]
                batch_labels[j] = labels[(i*batch_size)+j]

            yield batch_features, batch_labels

and when i call model.fit_generator(data_generator_plain(X_train, y_train, BATCH_SIZE), steps_per_epoch=X_train.shape[0]//BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(X_val, y_val)) the val_accuracy ends around 73% while the basic model.fit(X_train, y_train, epochs=NUM_EPOCHS, batch_size=BATCH_SIZE, validation_data=(X_val, y_val)) val_accuracy ends around 93%.

Is my generator implementation correct? In theory, it should perform the same as the built-in fit function, right? I imagine it is not showing the algorithm all the examples it needs to. Thank you!

  • $\begingroup$ I cannot spot a big problem. Maybe because fit has argument shuffle=True by default. If X_train.shape[0] is not a multiple of BATCH_SIZE, it would be another small discrepancy. Or maybe there is a bug in some other part of your code. You can always insert some print statements and manually debug such problems. $\endgroup$
    – Valentas
    May 21 '19 at 5:18

The Keras fit_generator takes in a python generator as an input to train the model over an array of training data. Usually, these generators are formed using built-in Keras functions such as ImageDataGenerator that takes in the image array as inputs and performs user-defined operations (such as augmentations, normalization, transformations, etc.) on the same in real time. This considerably reduces the memory consumption.

For more, refer Keras image preprocessing docs

  • $\begingroup$ The .fit function also accepts generators $\endgroup$
    – Bananach
    Feb 16 '20 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.