I've got 15 different datasets at about 10GB each. Each dataset comes with a binary 2D ground truth (10486147ish, 1) that I pull from it. I'm trying to figure out how to load each dataset, split them all with scikitlearn's train_test_split, then iterate over all 15 datasets per epoch. Under normal circumstances, the datasets would be shuffled as well, but I cannot figure out how to even do that since the data is too large to load all at once to shuffle (as such shuffling them is on the back burner for now).

Here's what my code looks like for one dataset.

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import sequence
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection imporrt train_test_split

arr = np.load ('source/dir/dataset1.npy', allow_pickle = True, fix_imports = True)
arr[arr == -inf = -9999]
rehape = arr.reshape(((arr.shape[0])*(arr.shape[1])), (arr.shape[2]))
drop = reshape[~np.all(reshape == -9999, axis = 1)]
#additional work done with -9999 here
truth = drop[:,46]
data = drop[:,0:45]

#callbacks deleted in code sample

encoder = LabelEncoder()
Y = encoder.transform(truth)
Y = Y.reshape(10486147, 1)
X = data.reshape(10486147, 45, 3)

seed = 7
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = seed)

model = sequential()
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3)))
model.add(Dense(1, kernel_initializer = 'normal', activation = 'sigmoid'))

model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 500, batch_size = 1000, callbacks = [deleted callbacks])

So that makes sense for one dataset, but as I've said before I've got 15 datasets to iterate through, and I don't think retraining on new data is the right step. Is there a way through dataset1.npy through dataset15.npy while properly splitting the ground truth as well.

Any suggestions?

  • $\begingroup$ Is an interesting problem. A possibility is to train models by parts and then summarize them. Some models allow this separation. $\endgroup$ Apr 27, 2019 at 23:34
  • $\begingroup$ I didn't understand if you'll use them all at the same time or each dataset has a different usage? but if your datasets have the same shapes, you can gether them into one dataset $\endgroup$
    – Kahina
    Apr 27, 2019 at 23:36
  • $\begingroup$ The one dataset will become too large to load into memory (about 150GB). $\endgroup$ Apr 28, 2019 at 7:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.