I am trying to create a prediction API using keras which loads the model predicts and closes the model. But initializing time in python is about 3-5 secs so each request takes around 5 secs to return the prediction irrespective of number inputs rows(predictions)
Is there any way to keep the model loaded and then stream the input data to get prediction. Like a Pre loaded model either through a socket or a port.
Similar to open office document converter
\program\soffice.exe -accept="socket,host=127.0.0.1,port=8100;urp;" -headless -nofirststartwizard -nologo
Keras Prediction Code
#!/usr/bin/env python3.6
import sys
import pandas as pd
from keras.models import load_model
model = load_model('model.h5')
X = pd.read_csv(sys.argv[1]).values
prediction = model.predict(X)
pd.DataFrame(prediction).to_json(sys.argv[2])
Script is called as
python3.6 predict.py input_scaled.csv output_scaled.json
The prediction time are as follows
#row time
1 4.76 secs
10 4.49 secs
50 5.37 secs
5000 5.46 secs
50000 12.7 secs