I have a network which takes a 256x256x128 input image which has a bunch of 3D convolutional and 3D transpose convolutional layers and ends up as a 128x128x64 reconstructed image.
And it's working reasonably well. But in trying to improve it, I'm butting up against a lack of fundamental understanding of convolutions.
I would like the network to be able to "cross reference", say, an area in the top left with an area in the bottom right to aid in its reconstruction. But as the network is purely convolutions (and batchnorms and relus), and the smallest layer is only 128,128,64, and the kernel size is max 3,3,3; I'm presuming that the network never gets the chance to look at the "whole" input image for context.
If I'm right about this, is there a standard way to enable this? For example a fully connected layer in the middle of the network?