How to make predictions of multiple input samples at once in tf 2 with keras

Question

I am quite confused on the output of model.predict

when after training I validate my model on around 6000 samples I use the following pseudo code:

model.fit(...)
predictions = model.predict(val_set)
len(predictions) # == len(val_set)

result: tensor array of shape=(len(tensor_array),14) (one prediction for each input sample)

in production I currently use the following code to ocr image numbers:

model = tf.keras.models.load_model('number_ocr_v2')
def predict(list_images):
    global model
    print("length:")
    print(len(list_images))
    #predictions = model.predict(list_images) # <- Same result
    predictions = model.predict_on_batch(list_images)
    print(len(predictions))
    print(predictions)

console Output:

length:
30
{...tf loading text...}
1
tf.Tensor([[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]], shape=(1, 14), dtype=float32)

(one prediction for x input samples)

Apparently the Model delivers one array with only the prediction of the first input tensor.

Where are the rest of the 30 Input Tensors? What is the difference between my training code and my production environment that explains why training code delivers predictions[:] and production code only delivers predictions[0]?

A Time consumption benchmark suggests that all 30 digits are beeing processed. Timing fits with 30 Predictions a ~70us per image

Best regards,

Julian

Answer 1

In my case, the Input Tensor Array was of length 30 but faulty constructed.

Make sure your inputs are right, otherwise it can lead to inaccurate results