Trained Tensorflow model performs poorly on inference

Question

I trained an image classification model using Keras with Tensorflow backend. The model got good accuracy on validation dataset as well as on the testing data, I save the entire model to .h5 format, here is my checkpoint callback.

checkpoint = ModelCheckpoint(model_name+".h5", monitor='val_acc', verbose=1, save_best_only=True, save_weights_only=False, mode='auto', period=1).

As I was hoping to use this model on Android so I refreeze the model to binary protobuf (.pb) using keras_to_tensorflow.

When performing inference using the model on mobile I noticed the model gives very wrong and random predictions. I have tried exploring other reasons why this could be the case like I found here still, it seems clear the issue is not with loading the images.

Also, performing inference with the converted model on Tensorflow Python still gives the same wrong/random predictions. Here is my code for performing inference in Python.

def model_predict( model_path, image_path, model_input, model_output, class_names ):

    with tf.Graph().as_default() as graph: # Set default graph as graph

        with tf.Session() as sess:
            # Load the graph in graph_def
            print("load graph")

            # We load the protobuf file from the disk and parse it to retrive the unserialized graph_drf
            with gfile.FastGFile(model_path,'rb') as f:

                print("Load Image...")
                # Read the image & get statstics
                np_image = Image.open(image_path)
                np_image = np.array(np_image).astype('float32')/255
                np_image = np.resize(np_image, (224, 224, 3))
                np_image = np.expand_dims(np_image, axis=0)


                # Set FCN graph to the default graph
                graph_def = tf.GraphDef()
                graph_def.ParseFromString(f.read())
                sess.graph.as_default()

                # Import a graph_def into the current default Graph (In this case, the weights are (typically) embedded in the graph)

                tf.import_graph_def(
                graph_def,
                input_map=None,
                return_elements=None,
                name="",
                op_dict=None,
                producer_op_list=None
                )

                # INFERENCE Here
                m_input = graph.get_tensor_by_name(model_input) # Input Tensor
                m_output = graph.get_tensor_by_name(model_output) # Output Tensor

                print ("Shape of input : ", tf.shape(m_input))
                #initialize_all_variables
                tf.global_variables_initializer()

                # Run model on single image
                Session_out = sess.run( m_output, feed_dict = {m_input : np_image} )

                print("Predicted class:", class_names[Session_out[0].argmax()] )

How do I perform inference using Tensorflow Python/Android with a save .pb model?

Others have suggested I save the session used for training and load them to Tensorflow when performing inference. If this is the case how do I load the saved session in Tensorflow android?

I am sure the model did not overfit, it performs very well when using Keras.

Answer 1

I believe the samples that you are using for inference belongs to a different distribution of data from training and test set. So, I would suggest you to verify if the above case is true and if it is then you should fine-tune your model for the examples that you are trying to carry out inference.

Answer 2

See the model must be overfitting as the prediction code looks correct. Kindly check for other metrices than only using accuracy. Print the confidence matrix and see the results.