Reading the ResNet paper, paragraph 3.3:
The convolutional layers mostly have 3×3 filters and follow two simple design rules: (i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer.
The network uses convolutional layers with stride 2 to halve the feature map size, but this divides both the width and the height of the feature maps, so the total area is divided by 4, not by 2.
So I am guessing the time complexity per layer is halved every time the feature maps are halved. Did I miss something?