Let's say that I have a visual encoder (CNN or ViT) which outputs a volume of local features of dimensions WxHxD plus a global feature vector of dimension D, which I'm currently using as the backbone of multiple downstream classification and text generation tasks (concurrently, as in multitask learning). How easy/hard would it be to append yet another downstream task to the list, namely, an object detection task (with bounding boxes)?
For example, let's say that my best backbone so far is a DenseNet121. Is there an easy to way to plug in an object detector on top of it and train the whole thing end-to-end, along with all the other downstream tasks at the same time?
I'm interested in ideas to get this done in Pytorch. I was thinking that maybe a Faster R-CNN might work, but I'm not really sure. I have zero prior experience doing object detection (this would be the first time), and Faster R-CNN is relatively old (2015), so for sure there must be better architectures/algorithms that have come out since then.