gpu pytorch code way slower than cpu code?

Question

I have the following pytorch code in a jupyter notebook:

import torch
t_cpu = torch.rand(500,500,500)
%timeit t_cpu @ t_cpu

Which outputs:

422 ms ± 3.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

And the following code, which took like 100 times as long:

import torch
t_gpu = torch.rand(500,500,500).cuda()
%timeit t_gpu @ t_gpu

which outputs:

31.8 ms ± 550 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Clearly, the reason the gpu version took longer is because it did 1000 loops each.

Why did it do 1000 loops each, given that I didn't specify this at all?

ps. side question: what does the "@ t_cpu" do in "%timeit t_cpu @ t_cpu"?

Answer 1

TL;DR

GPU runs faster than CPU (31.8ms < 422ms).

Your results basically say: "The average run time of your CPU statement is 422ms and the average run time of your GPU statement is 31.8ms".

The second experiment runs 1000 times because you didn't specify it at all. If you check the documentation, it says:

-n: execute the given statement times in a loop. If is not provided, is determined so as to get sufficient accuracy.

Number of runs

The %timeit function estimates the run time of a statement. It does that by executing the statement multiple times and measuring the average run time of it.

The number of times it runs the experiment is $N\times{R}$, where $N$ is the number of loops and $R$ is the number of times it repeats it.

If you do not specify $N$ and $R$, the default $R$ is $7$ and $N$ may vary. In your case, you did not specify any. So, in the first experiment, you had $R=7$ and $N=1$. The second experiment was $R=7$ and $N=1000$.

$N$ varies depending on how long it takes to run a single loop. If your statement is time-consuming, %timeit will execute fewer loops, whereas fast statements are executed more often. This is the case because it will give a better accuracy of the actual run time of your statement.

Results

You should not consider the time to run a statement based on the time that it takes to run the %timeit function. As you noticed, the number of loops may vary. Instead, try to interpret the printout.

For example, you had:

422 ms ± 3.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

422 ms ± 3.93 ms per loop: this is the mean average time and standard deviation it takes to run your statement one time. A loop is a single execution of your statement. Your statement is what you have defined when calling the function (in your case it is t_cpu ).

of 7 runs, 1 loop each: this is the number of times that %timeit ran your statement to calculated the average execution time. If you think that running the t_cpu only $7$ times is not reliable enough to estimate the $422ms$ then you may increase $N$ and $R$.

Documentation

I really don't mind answering your questions here. However, if you want faster clarifications to your answers, a good resource to read is usually the documentation of the function itself.

The documentation of the %timeit function can be found here.