How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

Question

As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and still don't) have the hardware to fully train models from scratch.

Lately, I've been dealing with more and more images, and the need to feed data on the fly during training (fine-tuning) has become important. My current setup is essentially the legacy train.py script for training and feed_dict method for inference.

I've been reading a lot on Estimators API and their usual pipeline, but I just cannot find tutorials or help on how to use pretrained models like I have with that pipeline; all examples work on image classification and do not mix dataset creation and model training, which is a real puzzle.

_{*Tensorflow r1.14}

---

Step 1: Dataset creation

So, given my data in the following format:

data = [ 
  [img_path_1, boxes_1],
  [img_path_2, boxes_2],
  ...,
  [img_path_N, boxes_N]
]

Where each boxes_i is in the format:

[
  [label_A, xa, ya, xb, xb],
  [label_B, xa, ya, xb, xb],
  [label_A, xa, ya, xb, xb],
  ...,
  [label_D, xa, ya, xb, xb],
]

According to some tutorial and tweaked for object detection instead of classification, it seems like I would need to process images like this (further preprocessing and batching functions left out for simplicity):

dataset = tf.data.Dataset.from_tensor_slices(data)

def load_image_and_annotations(path_and_boxes):
    path, boxes = path_and_boxes
    image_string = tf.read_file(path)

    image = tf.image.decode_jpeg(image_string, channels=3)
    image = tf.image.convert_image_dtype(image, tf.float32)
    return (image, boxes)

dataset = dataset.map(load_image_and_annotations, num_parallel_calls=...)

and then make use of the tf.data.Iterator object to traverse the dataset as needed.

Step 2: Training

Here is mostly where I'm unsure of how to proceed. The pre-made estimators and custom estimators tutorials seem inappropriate for my task since I already have a frozen model that I wish to convert to an estimator (instead of fully building one).

---

How should I proceed to transition from my old setup to this new pipeline ?

Answer 1

After a lot of research, I came to the conclusion that this is doable, but would require a lot of work.

Essentially, the steps to achieve such a transformation are as follows:

1. Use this version of the model exporter (or any other solution) to re-export the savedmodel with its variables (by default, the Model Zoo does not give the variables, only an inference-ready model).

2. Load the model using the SavedModel API, i.e. loaded = tf.saved_model.load(...).

3. Extract the inference function: inference_func = loaded.signatures['serving_default'].
4. Define your loss function. This part is the most complex because the existing functions are part of the Object Detection API which is pretty much outdated. Tensorflow is already working on the upgrade of the API but at the time of writing this, they still expect a couple more months to finish it.
5. Set up your Dataset.
6. Train your model.

The best option is probably to wait for the new release of the official OD API.