I am building a ResNet. I have two separate blocks: Cnn block, Reduce block. Cnn block - 1 cnn layer, activation, Batch Normal -> 1 cnn, activation, Batch Normal, so 2 CNN in this block. In Reduce block, 1 cnn, activation, Batch Normal, avg pooling, 1 cnn, activation, Batch Normal.
I am wondering if we should include Residual in both the Cnn block and Reduce block? or just for Cnn block? From my experiment, including Residual only in Cnn block is giving slightly better accuracy on the validation set. I am not able to figure out the reason why.