I’m looking at doing something like semantic segmentation of images but where I only have pretty coarse-grained labels - roughly, for each 32x32 patch, I know if the answer should be “yes”, “no” or “unknown”.
I’d like to start with a pretrained net that is reasonably well suited for this, and then modify it.
This is a bit like FCN-ResNet101 in the torch hub, but I don’t need any of the stuff that converts the feature maps back up to full resolution - and in my case it would probably both be slow and hurt performance since my labels just aren’t that exact.
I’m not sure what the ideal size of the receptive field for each position of the feature map should be, but I suspect it should be a bit on the small side, around 128x128 to 256x256, based on what I know about the data. Receptive fields for resnet/resnext-50 and 101 before the top pooling layer are really really huge by comparison, 483 or 1027 square (most of which comes from padding with typical 224x224 size inputs). In my case I’d like to feed inputs 1024x1024 or larger and have receptive fields not larger than 256x256 with a 32x32 stride - ideally :)
Is there any example in the literature or a well-known dataset of coarse segmentation similar to this? What would be the state of the art?
What is the best backbone network to use as a starting point? Can I use a pretrained classifier network (I rather like the various swsl_resnext models in timm) and just remove the top pooling layer? Is any specific classifier best for this? (resnext vs efficientnet vs...) Or, should I use a network already intended for segmentation? (in that case I’m a lot less clear on what to remove...)
If I use a network with a large receptive field, how should I shrink it? Should I just remove blocks from the top until the field shrinks enough?