Universal Sentence Encoder MultiLingual inside Flask

Question

I am having an issue using Universal Sentence Encoder(USE) Multilingual inside a flask app. I am using it with TF2.

When I initialize the flask app and load the USE model using hub.load() and then call it on text to get the embeddings, it works perfectly fine.

But when I use that same object inside a request, it does not work. I see no errors coming and after some time the flask app sends a connection error.

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Correction: All things works fine when launching flask alone. But when I wrap the flask app using uwsgi, I face the problem as mentioned above.

uwsgi --http :5000 --logger file:logfile=/tmp/uwsgi.log,maxsize=1000000000,backupname=/tmp/uwsgi.log.rotated --manage-script-name --wsgi-file main-app.py --callable app --processes 2 --enable-threads --listen 128 &

Let me know if anyone has faced similar issue before.

Answer 1

I've had a similar issue but with App Engine / GCloud.

Universal Sentence Encoder(USE) Multilingual requires lots of CPU and RAM. It could be a problem using uwsgi.

First, you should see how many RAM and CPU is used using flask only (see the related Python process in the task management monitor), and then see if the same app reach a memory limit with uwsgi that explains this error.

Then, it is possible that uwsgi has parameters to limit the use of CPU or RAM to share them. You should check if the limits are enough for the Universal Sentence Encoder(USE) Multilingual.

Note that uwsgi uses probably more memory than the flask application only.

It could also be due to a timeout parameter that could explain the error. The request's timeout could be increased to let the USE complete its job.