1
$\begingroup$

I'm trying to train a model which in my opinion is taking too long compared to other datasets given that it's taking about 9s to complete a step. I think that the problem is because the dataset is not being stored on ram, but I'm not sure of this.

The code is the following:

def load_data():

    train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
    train_generator = train_datagen.flow_from_directory(path1, target_size=(200, 200), batch_size=32, class_mode="binary")

    test_datagen = ImageDataGenerator(rescale=1./255)
    test_generator = test_datagen.flow_from_directory(path2, target_size=(200, 200),batch_size=32, class_mode="binary")

    return train_generator, test_generator

Model:

  • Sequential model
  • 2 Convolutional layers with 32 neurons, activation = relu.
  • 1 Convolutional layer with 64 neurons, activation = relu.
  • Flattening and Dense layer, activation = relu.
  • Dropout of 0.5
  • Output layer (Dense) with sigmoid activation.
  • Adam optimizer.
  • Loss: binary cross entropy.

Fit:

model.fit_generator(x, steps_per_epoch=37, epochs=50, validation_data=y, validation_steps=3, callbacks=[tensorboard])
  • My dataset has 1201 images and 2 classes.
  • I built the model following this tutorial.
  • My GPU is a GTX 1060 3gb.
  • 8gb of ram.
  • The images are being reshaped to 200x200.

If you could help me I'd appreciate it. Thank you very much!

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Which datasets are you using? how long is "too long"? Are you checking that your GPU is actually being used? $\endgroup$
    – Juan Antonio Gomez Moriano
    Jan 1, 2019 at 21:34
  • $\begingroup$ I'm using my own dataset, it takes about 9s per step to complete and I've checked that my GPU is being used. $\endgroup$
    – Santiago Pardal
    Jan 1, 2019 at 23:25
  • $\begingroup$ So the 9s is per step, not per epoch, right? Are you really sure you are using the GPU? try to remove the code that forces the GPU to be used and run it again, and lets see how it goes. $\endgroup$
    – Juan Antonio Gomez Moriano
    Jan 3, 2019 at 3:37
  • $\begingroup$ @JuanAntonioGomezMoriano Yes, I'm sure it's being used because the GPU gets hotter as soon as the program starts and I've checked with nvprof. $\endgroup$
    – Santiago Pardal
    Jan 3, 2019 at 15:00
  • $\begingroup$ Something that might help would be to increase the size of your batches, you normally want to use ALL of your GPU memory, so the GPU itself can be busy doing calculations instead of just reading the images. This actually bite me a couple of times, with images of 200x200 you can possibly use batches of 128. Another option would be to try to use larger images. $\endgroup$
    – Juan Antonio Gomez Moriano
    Jan 3, 2019 at 21:18

1 Answer 1

Reset to default
0
$\begingroup$

Your images are probably too large for using just three convolutions. So the number of parameters is huge during flattening (and the net cannot learn global features of the images, but this is a different problem).

Try to

  • use more convolution and pooling operations

  • see if strided convolutions help

  • check the number of parameters using the summary method of Keras

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.