Why is training take so long on my GPU?

Question

Details:

GPU: GTX 1080

Training: ~1.1 Million images belonging to 10 classes

Validation: ~150 Thousand images belonging to 10 classes

Time per Epoch: ~10 hours

I've setup CUDA, cuDNN and Tensorflow( Tensorflow GPU as well).

I don't think my model is that complicated that is takes 10 hours per epoch. I even checked if my GPU was the problem but it wasn't.

Is the training time due to the Fully connected layers?

My model:

model = Sequential()
model.add()
model.add(Conv2D(64, (3, 3), padding="same", strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding="same", strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Flatten())
model.add(Dense(256))
model.add(Activation('relu'))
model.add(Dense(4096))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.summary()

opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)

model.compile(loss='categorical_crossentropy',
          optimizer=opt,
          metrics=['accuracy']
          )

Because there is a lot of data I used the ImageDataGenerator.

gen = ImageDataGenerator(
 horizontal_flip=True
)

train_gen = gen.flow_from_directory(
        'train/',
        target_size=(512, 512),
        batch_size=5,
        class_mode="categorical"
)

valid_gen = gen.flow_from_directory(
        'validation/',
        target_size=(512, 512),
        batch_size=5,
        class_mode="categorical"
)

Answer 1

That's about expected. If you divide the number of seconds by the number of images you processed, you get 33 milliseconds per image, which seems about right for such a small network. Larger networks usually take in the ballpark of 50 to 200 milliseconds per image.

Yes, a large dense layer is likely to hurt your performance, since that's a huge matrix (256 by 4096) and a large matrix multiplication to go along with it every time you run the network.

Answer 2

As shimao said, that's about what you'd expect. Despite not having many layers, an input size of 512x512 is a large image to be convolving over. The large computation time is likely more due to convolving 64 filters over the large image, instead of the fully connected layers.

The network you've put together has a funny information bottleneck in it though. You start off with 64 filters on the original sized image, only decreasing as your image size reduces. As the image passes through your network, the features you're learning become more and more abstracted and complex. Your Conv2D(32, (3, 3)) layer essentially limits the network to learning a 128x128 map of 32 features.

Most network architectures double the number of features every time they pool, and most recent imagenet architectures actually ditch the fully connected layers in favour of an average pool over the final feature map, and basically perform logistic regression on the output of that pool.

Try starting off with fewer filters, say 16 in your first convolution layer, doubling each time you stride or pool. Do this a few more times than you are, to increase receptive field and decrease feature map size. Do this down to 64x64 or 32x32, which would be 128 or 256 filters. You can use Keras' Global Avg or Max pooling to eliminate fully connected layers as well. That should about double the speed of the network, and I would expect an increase in accuracy at the same time.