enter image description here

So trainfiles is a list that contains the files' directory and name e.g. ['../train/1.npy' , '../train/2.npy'] and then I create a dataset as shown in the middle of the code then I apply it to model fit function

just adding shuffle=True in the argument for model fit function doesn't do anything it seems. How do I go about shuffling the trainfiles correctly? i.e. when does the trainGenerator function gets executed? i.e. does it gets executed 516 times when generating 1 batch?

can I just add random.shuffle(trainfiles) right above the for-loop in trainGenerator?

  • $\begingroup$ Typically the generator generates one batch per call to next(). How many files do you have? I'm wondering if the generator approach is actually not the best one. $\endgroup$
    – Matthew
    May 7, 2020 at 0:53
  • $\begingroup$ I'm dealing with over 75000 npy files. I tried concatenating all the files into one big npy and i get Out Of Memory problem. $\endgroup$
    – Kevin Kim
    May 7, 2020 at 3:11
  • $\begingroup$ @Matthew do you think i can just add random.shuffle(trainfiles) right above the for-loop? $\endgroup$
    – Kevin Kim
    May 7, 2020 at 3:12

1 Answer 1


The big issue here is that your generator yields after each file is loaded. This means that your batch size is always the number of examples stored in each file and that the training examples in each batch are always the same.

You can absolutely shuffle the order of the files, but that's just shuffling the batch order around and not shuffling data examples among batches, which is very important for the optimizer. (Each batch yields an estimate of the gradient of the loss function with respect to the model parameters given the particular batch data. The more combinations of examples you're able to give the network, the better you're able to estimate the gradient with respect to the training set.) There are lots of ways to try to fix this, the Tensorflow docs have tutorials for some of them. (You could load a random set of files and deliver batches that contain their data intermixed, then load more, etc)

Also, you have 75,000 files that appear to be (40, 40, 15). That means the whole dataset is (3 000 000, 40, 15) if all the files are the same size, which is ~14.4 GB if you're storing the data as np.float64. That's big, but not ridiculous. You could probably get a significant chunk of that in memory at once, which is probably better.

Is this float data? If so, do you need that level of precision (np.float64, the default)? I'm guessing probably not. If you can store it as np.float32 or even np.float16, you might be able to just get the whole thing in memory. If not, you could load it as bigger chunks (half the dataset at once, maybe) and intermix examples among them. Just something to think about.


Also, if you're going to concatenate them all, don't do it in a loop with a big array and iteratively concatenate the smaller arrays:

# NO

big_array = np.load("first_train_file.npy")

for train_file in rest_of_train_files:
    tmp_file = np.load(train_file)
    np.concat((big_array, tmp_file), axis=0)


list_of_little_arrays = []

for train_file in all_train_files:
    tmp_file = np.load(train_file)

all_data = np.concatenate(list_of_little_arrays)

You're going to have loads of memory problems using the first way. I don't know that that's what you tried or not, but just in case.

  • $\begingroup$ As for shuffling the training examples among batches, i.e. each batch isn't going to have the same training examples, I've tried that, but the training seems too sporadic. I do understand that the more combinations of examples I give to the NN, it should generalize better, but the loss fluctuates too much than usual. Another surprising thing about that is if the validation score is say 1.6, then on the test score, it's actually 2.1 or something. Before shuffling the examples among batches, the val score was somewhat close to the test score tho $\endgroup$
    – Kevin Kim
    May 11, 2020 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.