Why the first prediction of neural network in PyTorch is slower than following predictions?

Question

So I have ResNet50 trained to classify images.

For each prediction I track the time needed for it (input and model are moved to GPU):

    start = time.time()
    result = self.model.forward(transformed_image)
    end = time.time()
    print(end - start)

And always I get the following output:

1.0592937469482422
0.05996203422546387
0.06096029281616211
0.04996800422668457

So the first prediction is ~20 times longer than the following ones.

Why? And what happens behind the scenes when we launch prediction for the first time, using Torch?

Answer 1

I have seen a similar question several times before. See https://stackoverflow.com/a/55577921/9212382 for a possible explanation.

Answer 2

This is due to the fact that first time is very important, everything is set up during the first pass like cache, memory on the GPU, graph optimization, etc. Also 1 seconds is not that long it can take few seconds just to run matrix multiplication on GPU