1
$\begingroup$

I have data that I read from feather file, and it's divided into x and y, where y is the label for the data. x is DataFrame and y is DataFrame column.

The dataset is huge and I'm looking for a way to fit it into the memory of the GPU.

I was trying to run model.fit with limiting just the batch_size, but it didn't work, so I have found by experimentation that I also need to limit the size of x and y for it to work. The actual size with which the learning can proceed is count=int(x.shape[0]/100)

One approach is to randomly select count items from both x and y, and repeatedly run model.fit with that data. It would look like this:

def get_sample_seletor(total_count,requested_count):
    import random
    result=list(range(0,total_count))
    random.shuffle(result)
       
    return result[0:requested_count]

for i in range(100):
    count=int(x.shape[0]/100)
    sample=get_sample_seletor(x.shape[0],count)
    x_w=x.iloc[sample]
    y_w=y[sample]
    model.fit(x_w,y_w,batch_size=count)

However, this carries performance penalty since we would have to call model.fit 100 times for every training epoch.

Can the gpu training be done without having to split the data before calling model.fit?

$\endgroup$
1
  • $\begingroup$ Have a look at data generators, which can load the data into memory as needed instead of having to load all the data into memory at once. $\endgroup$
    – Oxbowerce
    Feb 5, 2023 at 20:07

1 Answer 1

1
$\begingroup$

Normally you don't have a complete dataset on the GPU memory. Instead, you train in "mini-batches" sampled randomly from your dataset and only keep the current mini-batch on the GPU memory.

It's like what you coded, but it is done automatically by model.fit.

You can specify the batch size (the equivalent of your total_count) as an argument to model.fit. By default, it is 32 (check the docs).

This way, by default, if your entire dataset fits on the CPU RAM, as it seems, you should not need to do anything else to have the minibatch training to take place, just invoke model.fit with all the data and it will take care of it.

HOWEVER, I think you are facing this Keras/tensorflow bug affecting versions 2.6-2.10, which loads the whole dataset on GPU if it's a numpy array. I suggest trying with a not affected version.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.