0
$\begingroup$

I have a network which takes a 256x256x128 input image which has a bunch of 3D convolutional and 3D transpose convolutional layers and ends up as a 128x128x64 reconstructed image.

And it's working reasonably well. But in trying to improve it, I'm butting up against a lack of fundamental understanding of convolutions.

I would like the network to be able to "cross reference", say, an area in the top left with an area in the bottom right to aid in its reconstruction. But as the network is purely convolutions (and batchnorms and relus), and the smallest layer is only 128,128,64, and the kernel size is max 3,3,3; I'm presuming that the network never gets the chance to look at the "whole" input image for context.

If I'm right about this, is there a standard way to enable this? For example a fully connected layer in the middle of the network?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

the network never gets the chance to look at the "whole" input image

Generally, incorrect. The "bunch of 3D convolutional and 3D transpose convolutional layers" (if setup correctly) got to see the whole image.

Internally, a kernel swipes through the whole image region-by-region, with no part of the image missed. The convolutional layers etc. then somehow 'mix' the details together to give the output.

Take for example a classification problem, there are methods e.g. Class Activation Mapping to check which part of the image the model is focusing on. You can see that the model focus on different parts (possibly > 1) of the image to give prediction.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for the information. I wasn't aware that the kernel was able to combine information without the fully connected layer which generally happens at the end of classification networks. $\endgroup$
    – Omroth
    Nov 4, 2022 at 10:39
  • $\begingroup$ The kernel is a part of convolutional layer, which concentrates on a subset of input; the "combine information" exercise is the effort of convolutional layer+ fully-connected layer + max-pooling etc. altogether. $\endgroup$
    – lpounng
    Nov 4, 2022 at 10:42
  • $\begingroup$ As analog, say we are looking at a huge picture. Our eyes ('kernel') first swipes through sub-regions of the picture piece-by-piece (e.g. there is a bird at top-right; a car at bottom-left...) and pass these info to our brain. The brain (composed of convolution/FC/max-pooling layers etc.) then combines all these info together to form the overall impression. $\endgroup$
    – lpounng
    Nov 4, 2022 at 10:47
  • $\begingroup$ Indeed - so I do need to put a fully connected layer in. $\endgroup$
    – Omroth
    Nov 4, 2022 at 10:47
  • $\begingroup$ Not necessarily - FC layer is not a must, though we see it in a lot of applications. Check this post. $\endgroup$
    – lpounng
    Nov 4, 2022 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.