How can I load data of shape:

X_train : (4864,6989)
X_test : (2085, 6989)
y_train : (4864, 270)
y_test : (2005, 270)

into the memory? I have 16 GB of RAM.

What I have tried:

  1. Loading data in chunks

    extracted_df = pd.read_csv("ExtractedData.csv", chunksize=6953)

  2. Using virtual memory (paging) by going in the system settings

  3. Reducing the chunk size.

I am able to fit the data on a relatively simple model. However for the model I have defined, I can't seem to get out of the memory error.

def build_model(embed_size, max_length, vocab_size):
    def build_model():
        model = Sequential()
        model.add(Embedding(vocab_size, embed_size, input_length=max_length))
        model.add(Conv1D(embed_size, 7, activation='tanh', padding='same'))
        model.add(Conv1D(embed_size, 7, activation='tanh', padding='same'))
        model.add(Dense(150, activation="tanh"))
        model.add(Dense(150, activation="tanh"))
        model.add(Dense(output_nodes, activation="softmax"))
        plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)
        return model

    return build_model

feature_nodes = X_train.shape[1]
estimator = KerasClassifier(build_fn=build_model(feature_nodes, feature_nodes, 1000), epochs=3, batch_size=16,
                                verbose=1, callbacks=[EarlyStopping(patience=3, monitor='accuracy')],
                                validation_data=(X_test, y_test))

I don't understand dask enough to try it but as I understand, it is for really huge data. My data has 98k rows and 2 columns. Any suggestions? Thanks.

  • 1
    $\begingroup$ Try looking at making a custom keras data generator to feed the data in batches to the model and then use to .fit_generator method to fit the model. $\endgroup$ – Oxbowerce Mar 28 at 12:23
  • $\begingroup$ Thank you. Is there a skeleton code that you can provide? $\endgroup$ – Anan Srivastava Mar 28 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.