2
$\begingroup$

When I call Keras' fit_generator(), passing in a custom generator class I created, I see "Epoch 1/1" in the spew and that's all. It hangs right there, and the generator is never called. I know this because I put print statements in getitem that are never printed.

This data generator is a modified version of Shervine Amidi's tutorial example of a generator that inherits from the Keras Sequence object:

class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, batchID, 
                 batch_size = 32, 
                 dim = (32,32,32)):

        self.dim = dim
        self.batch_size = batch_size
        self.datafile_IDs = []
        self.labelfile_IDs = []
        self.batchID = batchID    
        self.DataDir = "data/"
        self.BatchDir = ""

        DataDir = self.DataDir
        BatchDir = DataDir + batchID + "/"
        self.BatchDir = BatchDir

        path = BatchDir + "datafilenames_" + batchID + ".pkl"                                    
        fd = open(path, "rb")
        self.datafile_IDs = pkl.load(fd)
        fd.close()

        path = BatchDir + "labelfilenames_" + batchID + ".pkl"        
        fd = open(path, "rb")
        self.labelfile_IDs = pkl.load(fd)
        fd.close()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(
            np.floor(len(self.datafile_IDs) / self.batch_size))

    def __getitem__(self, index):

        'Generate one batch of data'

        datafn = self.datafile_IDs[index]        
        labelfn = self.labelfile_IDs[index]  

        print("In getitem: index = %d, datafn = %s, labelfn = %s" % (
               index, datafn, labelfn))

        batch_size = self.batch_size

        # Initialize data arrays for this batch     
        X = np.empty((self.batch_size, *self.dim))
        y = np.empty((self.batch_size), dtype=int)         

        BatchDir = self.BatchDir

        # Load data
        datafn = BatchDir + datafn
        X = np.load(datafn)

        # Load label
        labelfn = BatchDir + labelfn
        y = np.load(labelfn)

        return X, y        

genbatchfiles(df_short, batchID = "short", batch_size = 20)
params = {'batchID': "short", 'batch_size': 20, 'dim': (100, 10088)} 
dg = DataGenerator(**params) 
time_series_length, input_dim, output_dim = 100, 10088, 1
model = Sequential()
model.add(LSTM(20, input_shape=(time_series_length, input_dim))) 
model.add(Dense(output_dim, activation='relu'))

model.compile(loss='mean_squared_error',
              optimizer='sgd',
              metrics=['accuracy'])
model.fit_generator(generator = dg,
                    steps_per_epoch = 5,
                    use_multiprocessing = True, 
                    workers = 6,
                    verbose = 2)

Per the request of the person who responded below, I'm adding the code for the function genbatchfiles that I use to generate the input data:

def genbatchfiles(df, batchID, batch_size = 5):

    DataDir = "data/"
    BatchDir = DataDir + batchID + "/"

    # If directory does not already exist
    # for this batch, create it.
    if not os.path.isdir(BatchDir):
        mkdir(BatchDir)

    # Column "sigccm" (signal cross correlation matrix) holds
    # a series each of whose elements is a 100 X 10088 array.
    # Column "Attack" holds labels with values 1's and 0's that
    # indicate whether a cell phone spoofing signal is present
    sigccm = df["sigccm"]
    attack = df["Attack"]

    idcnt = 0
    datafiles = []
    labelfiles = []

    # number of records to process
    nrecs = len(sigccm)

    for i in range(0, nrecs, batch_size):

        if i + batch_size > nrecs:
            upperbound = nrecs 
        else: 
            upperbound = i + batch_size

        x = np.stack(sigccm[i : upperbound]) 
        y = np.stack(attack[i : upperbound]) 

        fnm = 'data_{:s}{:02d}.npy'.format(batchID, idcnt)
        datafiles.append(fnm)        
        np.save(BatchDir + fnm, x)

        fnm = 'labels_{:s}{:02d}.npy'.format(batchID, idcnt)
        labelfiles.append(fnm)        
        np.save(BatchDir + fnm, y)

        idcnt = idcnt + 1

    path = BatchDir + "datafilenames_" + batchID + ".pkl"

    fd = open(path, "wb")
    pkl.dump(datafiles, fd)
    fd.close() 

    path = BatchDir + "labelfilenames_" + batchID + ".pkl"

    fd = open(path, "wb")
    pkl.dump(labelfiles, fd)
    fd.close()    

Here's how you can generate some phony data that has the same shape as the data I am feeding into my Keras LSTM:

x = np.random.rand(150, 100, 10088).tolist()
df = pd.DataFrame({"sigccm" : x})
bools = np.round(np.random.rand(150), decimals=0)
attack = pd.Series(bools)
df["Attack"] = attack

Followed by:

genbatchfiles(df, batchID = "phony", batch_size = 20)
params = {'batchID': "phony", 'batch_size': 20, 'dim': (100, 10088)} 
dg = DataGenerator(**params) 

Followed by the LSTM code.

When I feed this phony data into the LSTM, Keras hangs in the generator in exactly the same way that it hangs with the real data, so anyone who wishes to do so ought to be able to repro the bug I'm seeing.

Warning: Even this toy dataset is huge. Each record is about a megabyte, and the phony data I generate above has 150 records, and so it contains about 150MB.

$\endgroup$
  • $\begingroup$ please provide a definition of the genbatchfiles() function so we could reproduce your error $\endgroup$ – MaxU Feb 10 at 3:43
  • $\begingroup$ Done. Thanks for the rapid reply!!! $\endgroup$ – John Strong Feb 10 at 13:35

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.