1
$\begingroup$

I'm using BERT to transform text into 768 dim vector, It's multilingual :

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('paraphrase-multilingual-mpnet-base-v2')

Now i want to put the model into production but the embedding time is too much and i want to reduce and optimize the model to reduce the embedding time What are the libraries that enable me to do this ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

you can start by using torchscript, it may require changing ur whole code, and switching to transformers( by loading the backbone of the model and the last layers) so basically u get out from GIL interpreter, coz it does not support multithreading. by with torchscript u can run ur model in c++ env, there's also onnx which I believe it enhances performance.

if ur use case is not a real-time and you are using an API, you can use a queue mechanism like rabbitmq

Improve this answer
$\endgroup$
1
  • $\begingroup$ What is the difference between real time and an api ? "so basically u get out from GIL interpreter, coz it does not support multithreading" can you illustrate more what you mean ? @SADAK $\endgroup$
    – Mohy Mohamed
    Commented Jul 11, 2021 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.