0
$\begingroup$

I used torch.quantization.quantize_dynamic to reduce the model size but it is reducing my prediction Accuracy score.

I'm using that model file inside the Flask and doing some real time predictions, Due to the large size i'm facing issues while predicting. So could anyone please help me on reducing the bert model size using pytorch and guide me on who to do the real time predictions.

Improve this question
$\endgroup$
2
  • $\begingroup$ Due to the large size i'm facing issues while predicting. Can you describe those issues? $\endgroup$
    – noe
    Commented Jun 30, 2022 at 16:18
  • $\begingroup$ Thanks for the response, i faced issues like taking time for prediction, loading large size models in the flask, memory issues. $\endgroup$
    – Sujitha Chinnu
    Commented Jul 1, 2022 at 13:41

1 Answer 1

Reset to default
1
$\begingroup$

Are you reusing an existing Bert model or are you training it from scratch?

In all the cases, you can apply several solutions to your model:

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the response, I'm using bert_model = BertModel.from_pretrained('bert-base-uncased') and from that written the classifier and trained with my own data set. $\endgroup$
    – Sujitha Chinnu
    Commented Jul 1, 2022 at 13:44
  • $\begingroup$ Have you tried distill Bert? For information, if you train it from scratch only with your data, you can reduce its size considerably. $\endgroup$
    – Nicolas Martin
    Commented Jul 1, 2022 at 14:15
  • $\begingroup$ huggingface.co/distilbert-base-uncased $\endgroup$
    – Nicolas Martin
    Commented Jul 1, 2022 at 15:00
  • $\begingroup$ Note: I'm also developing a flask service with NLP models. I've written an article about it that might answers some questions about model optimization: medium.com/nerd-for-tech/… $\endgroup$
    – Nicolas Martin
    Commented Jul 2, 2022 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.