Can I train two stacked models end-to-end on different resolutions?

Question

Is it possible to stack two networks on top of each other that operate on different resolutions of input data?

So here's my usecase: like Google, I want to recognize text in images. Unlike Google, I have access to very limited computational resources. My solution for this is that instead of using a sequence-to-sequence network on the whole scene image, I first run object detection using YOLO and then pass the crop to a sequence-to-sequence model, Attention-OCR.

To further reduce processing time, I run the object detection on low resolution, and crop the detection result in the higher resolution input image, so I still have access to the high resolution input when using sequence recognition on my crop. To detect that there is text, the low resolution is adequate, but to read the text, the network needs the higher resolution input.

This all works fine, but I suspect I would get better performance if I could train the whole system end-to-end, so the crop gets directly optimized to result in the best reading of the text. I could stack the sequence-to-sequence model on top of the object detection model to do this, but then the text reading operates on the same low resolution input that is used for the text detection.

Does anyone have an idea how to solve this, or can anyone point me to research that is related to this?

Answer 1

I would suggest you to split your problem into two:

1. How to train?
2. How to make inference when resources are limited?

Very common pattern is to train model on large data (by renting AWS server, for instance) and then use its predictions to train much smaller network. Here is fundamental paper on the subject from Hinton et al.

This way you can stick to more conventional object detection method without stacking involved and then deploy 'squeezed' network to your device.

But back to your question. If you are rigid in your approach for various reasons and can't do like I described above, then lots of end-to-end approaches are available in literature, take a look at this to get started.

And even code.

I will just cite the paper:

A unified deep neural network, denoted the multi-scale CNN (MS-CNN), is proposed for fast multi-scale object detection. The MS- CNN consists of a proposal sub-network and a detection sub-network.

In the proposal sub-network, detection is performed at multiple output layers, so that receptive fields match objects of different scales. These complementary scale-specific detectors are combined to produce a strong multi-scale object detector. The unified network is learned end-to-end, by optimizing a multi-task loss.

Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs. State-of-the-art object detection performance, at up to 15 fps, is reported on datasets, such as KITTI and Caltech, containing a substantial number of small objects.

But nevertheless feel free to experiment with network compression (take a look at this). It typically has high degree of redundacy, and by being creative in exploring this redundacy it's possible to have many gains