I was reading this paper. The code is available in GitHub. In the README.md file, they've mentioned how to train the model

python prediction_train.py

with optional parameters.
Can anyone please explain how do I use this model to predict a video sequence? I'm new to deeplearning and tensorflow. So, I'm not able to understand the code properly. My current task is to just run the code and see the output (i.e. the videos predicted by this model).

All I could understand was, it uses tensirflow saver to save the checkpoints. I'm guessing these checkpoints are the intermediate trained model after few epochs (2000 in this case). How to use these models to predict the next frames of a video?

Any help is greatly appreciated :)

  • 1
    $\begingroup$ You can start by modifying the prediction_train.py into your own script. You need to load a pretrained model by providing the model path in line 48; then run the gen_images op instead of the train_op or summ_op to get the predicted images. $\endgroup$
    – user12075
    Jan 10, 2019 at 18:37

2 Answers 2


The TensorFlow saver is used to save the weights of a specific model at some given point. When you want to use a trained model, you must first define the model's architecture (which should be similar to the one used for saving the weights), then you can use the same "saver" class to restore the weights:

with tf.Session() as sess:
    # Restore variables from disk.
    saver.restore(sess, "../my_saved_model.ckpt")

Regarding your initial question. I think that if you are just starting with deep learning and TensorFlow, this is the wrong place to start and you should first understand how TensorFlow works in general by applying it at easier tasks like image classification (start with MNIST).

From what I understand, you need to use the "construct_model" function and pass it your initial image sequence (video) and some action tensor, and it should output a predicted frames:

def construct_model(images,
"""Build convolutional lstm video predictor using STP, CDNA, or DNA.
    images: tensor of ground truth image sequences
    actions: tensor of action sequences
    states: tensor of ground truth state sequences
    iter_num: tensor of the current training iteration (for sched. sampling)
    k: constant used for scheduled sampling. -1 to feed in own prediction.
    use_state: True to include state and action in prediction
    num_masks: the number of different pixel motion predictions (and
           the number of masks for each of those predictions)
    stp: True to use Spatial Transformer Predictor (STP)
    cdna: True to use Convoluational Dynamic Neural Advection (CDNA)
    dna: True to use Dynamic Neural Advection (DNA)
    context_frames: number of ground truth frames to pass in before
                feeding in own predictions
    gen_images: predicted future image frames
    gen_states: predicted future states
  • $\begingroup$ Thanks! And yes, I'm also parallelly reading neural networks used to classify MNIST digits from here: neuralnetworksanddeeplearning.com by Michael Nielsen. But again that takes some time. My guide has given me this task. So, I have to do this now :) $\endgroup$ Jan 11, 2019 at 5:52

The Prediction in video is too Easy..Use Opencv for video to read video into images and store them in an numpy array.Later you will load your tf model and then do some prediction .

Sections of this code were taken from:
import numpy as np

import os
import six.moves.urllib as urllib
import sys
import tarfile
import tensorflow as tf
import zipfile

from collections import defaultdict
from io import StringIO
from matplotlib import pyplot as plt
from PIL import Image

from utils import label_map_util

from utils import visualization_utils as vis_util

import cv2

# Path to frozen detection graph. This is the actual model that is used
# for the object detection.
PATH_TO_CKPT = '../freezed_pb5_optimized/frozen_inference_graph.pb'

# List of the strings that is used to add correct label for each box.
PATH_TO_LABELS = os.path.join('../../../training', 'object-detection.pbtxt')



def detect_in_video():

    # VideoWriter is the responsible of creating a copy of the video
    # used for the detections but with the detections overlays. Keep in
    # mind the frame size has to be the same as original video.
    out = cv2.VideoWriter('pikachu_detection_1v3.avi', cv2.VideoWriter_fourcc(
        'M', 'J', 'P', 'G'), 10, (1280, 720))

    detection_graph = tf.Graph()
    with detection_graph.as_default():
        od_graph_def = tf.GraphDef()
        with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
            serialized_graph = fid.read()
            tf.import_graph_def(od_graph_def, name='')

    label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
    categories = label_map_util.convert_label_map_to_categories(
        label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
    category_index = label_map_util.create_category_index(categories)

    with detection_graph.as_default():
        with tf.Session(graph=detection_graph) as sess:
            # Definite input and output Tensors for detection_graph
            image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
            # Each box represents a part of the image where a particular object
            # was detected.
            detection_boxes = detection_graph.get_tensor_by_name(
            # Each score represent how level of confidence for each of the objects.
            # Score is shown on the result image, together with the class
            # label.
            detection_scores = detection_graph.get_tensor_by_name(
            detection_classes = detection_graph.get_tensor_by_name(
            num_detections = detection_graph.get_tensor_by_name(
            cap = cv2.VideoCapture('PikachuKetchup.mp4')

                # Read the frame
                ret, frame = cap.read()

                # Recolor the frame. By default, OpenCV uses BGR color space.
                # This short blog post explains this better:
                # https://www.learnopencv.com/why-does-opencv-use-bgr-color-format/
                color_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

                image_np_expanded = np.expand_dims(color_frame, axis=0)

                # Actual detection.
                (boxes, scores, classes, num) = sess.run(
                    [detection_boxes, detection_scores,
                        detection_classes, num_detections],
                    feed_dict={image_tensor: image_np_expanded})

                # Visualization of the results of a detection.
                # note: perform the detections using a higher threshold

                cv2.imshow('frame', color_frame)
                output_rgb = cv2.cvtColor(color_frame, cv2.COLOR_RGB2BGR)

                if cv2.waitKey(1) & 0xFF == ord('q'):


def main():

if __name__ == '__main__':

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.