I've got 15 different datasets at about 10GB each. Each dataset comes with a binary 2D ground truth (10486147ish, 1) that I pull from it. I'm trying to figure out how to load each dataset, split them all with scikitlearn's train_test_split, then iterate over all 15 datasets per epoch. Under normal circumstances, the datasets would be shuffled as well, but I cannot figure out how to even do that since the data is too large to load all at once to shuffle (as such shuffling them is on the back burner for now).
Here's what my code looks like for one dataset.
import numpy as np import pandas as pd from keras.models import Sequential from keras.layers import Dense from keras.layers import LSTM from sklearn.preprocessing import sequence from sklearn.preprocessing import LabelEncoder from sklearn.model_selection imporrt train_test_split arr = np.load ('source/dir/dataset1.npy', allow_pickle = True, fix_imports = True) arr[arr == -inf = -9999] rehape = arr.reshape(((arr.shape)*(arr.shape)), (arr.shape)) drop = reshape[~np.all(reshape == -9999, axis = 1)] #additional work done with -9999 here truth = drop[:,46] data = drop[:,0:45] #callbacks deleted in code sample encoder = LabelEncoder() encoder.fit(truth) Y = encoder.transform(truth) Y = Y.reshape(10486147, 1) X = data.reshape(10486147, 45, 3) seed = 7 X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.33, random_state = seed) model = sequential() model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True)) model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3), return_sequences = True)) model.add(LSTM(units = 32, activation = relu, input_shape = (45, 3))) model.add(Dense(1, kernel_initializer = 'normal', activation = 'sigmoid')) model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy']) model.fit(X_train, y_train, validation_data = (X_test, y_test), epochs = 500, batch_size = 1000, callbacks = [deleted callbacks])
So that makes sense for one dataset, but as I've said before I've got 15 datasets to iterate through, and I don't think retraining on new data is the right step. Is there a way through dataset1.npy through dataset15.npy while properly splitting the ground truth as well.