I am trying to create a prediction API using keras which loads the model predicts and closes the model. But initializing time in python is about 3-5 secs so each request takes around 5 secs to return the prediction irrespective of number inputs rows(predictions)

Is there any way to keep the model loaded and then stream the input data to get prediction. Like a Pre loaded model either through a socket or a port.

Similar to open office document converter

\program\soffice.exe -accept="socket,host=,port=8100;urp;" -headless -nofirststartwizard -nologo

Keras Prediction Code

#!/usr/bin/env python3.6
import sys
import pandas as pd
from keras.models import load_model
model = load_model('model.h5')
X = pd.read_csv(sys.argv[1]).values
prediction = model.predict(X)

Script is called as

python3.6 predict.py input_scaled.csv output_scaled.json

The prediction time are as follows

#row    time
1       4.76 secs
10      4.49 secs
50      5.37 secs
5000    5.46 secs
50000   12.7 secs

I was able to get to work like this without flask or django. Just using default http.server in python

from http.server import BaseHTTPRequestHandler, HTTPServer
import logging
import sys
import pandas as pd
from keras.models import load_model
from urllib.parse import urlparse
model = load_model('model.h5')

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_header('Content-type', 'text/html')

    def do_GET(self):
        query = urlparse(self.path).query
        params = dict(qc.split("=") for qc in query.split("&"))
        X = pd.read_csv(params["input"]).values
        prediction = model.predict(X)

def run(server_class=HTTPServer, handler_class=S, port=8080):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    logging.info('Starting httpd...\n')
    except KeyboardInterrupt:
    logging.info('Stopping httpd...\n')

if __name__ == '__main__':
    from sys import argv

    if len(argv) == 2:

Trigger Server using

python3.6 predict_server.py 8000

API like


The easiest way I can think of is to create a flask app that will load the model once and have and endpoint where you can send your data as requests to your already loaded model.

A rough skeleton of the service would look like this:

from flask import Flask, request

app = Flask(__name__)

def index():
    return ''

@app.route('/predict/', methods=['GET', 'POST'])
def predict():
    X = pd.read_csv(request.get_data()).values
    prediction = model.predict(X)
    return pd.DataFrame(prediction).to_json()

if __name__ == "__main__":
    model = load_model('model.h5')

Then you could make an HTTP request to localhost:5000/predict, via another script and it would return your predictions, which then you could save or do whatever you want.

  • $\begingroup$ I do not use flask, but how do we know that the model in the predict controller is loaded only once during the lifecycle of the app (and not for example like php does loads it every time anew)? $\endgroup$
    – Nikos M.
    Jan 6 '21 at 10:26
  • $\begingroup$ My bad, I forgot to show where the model is loaded. This happens once, and remains loaded as long as the server is running. Every time a request is sent to the /predict endpoint, the model makes a prediction. $\endgroup$
    – Djib2011
    Jan 6 '21 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.