I've got 15 different datasets at about 10GB each. Each dataset comes with a binary 2D ground truth (10486147ish, 1) that I pull from it. I'm trying to figure out how to load each dataset, split them all with scikitlearn's train_test_split, then iterate over all 15 datasets per epoch. Under normal circumstances, the datasets would be shuffled as well, but I cannot figure out how to even do that since the data is too large to load all at once to shuffle (as such shuffling them is on the back burner for now).

Here's what my code looks like for one dataset.

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from sklearn.preprocessing import sequence
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection imporrt train_test_split

arr = np.load ('source/dir/dataset1.npy', allow_pickle = True, fix_imports = True)
arr[arr == -inf = -9999]
rehape = arr.reshape(((arr.shape[0])*(arr.shape[1])), (arr.shape[2]))
drop = reshape[~np.all(reshape == -9999, axis = 1)]
#additional work done with -9999 here
truth = drop[:,46]
data = drop[:,0:45]

#callbacks deleted in code sample

encoder = LabelEncoder()
Y = encoder.transform(truth)
Y = Y.reshape(10486147, 1)
X = data.reshape(10486147, 45, 3)

seed = 7
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = seed)

model = sequential()
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True))
model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3)))
model.add(Dense(1, kernel_initializer = 'normal', activation = 'sigmoid'))

model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 500, batch_size = 1000, callbacks = [deleted callbacks])

So that makes sense for one dataset, but as I've said before I've got 15 datasets to iterate through, and I don't think retraining on new data is the right step. Is there a way through dataset1.npy through dataset15.npy while properly splitting the ground truth as well.

Any suggestions?

  • $\begingroup$ Is an interesting problem. A possibility is to train models by parts and then summarize them. Some models allow this separation. $\endgroup$ – Juan Esteban de la Calle Apr 27 '19 at 23:34
  • $\begingroup$ I didn't understand if you'll use them all at the same time or each dataset has a different usage? but if your datasets have the same shapes, you can gether them into one dataset $\endgroup$ – Kahina Apr 27 '19 at 23:36
  • $\begingroup$ The one dataset will become too large to load into memory (about 150GB). $\endgroup$ – Hexadecimalism Apr 28 '19 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.