0
$\begingroup$

I used torch.quantization.quantize_dynamic to reduce the model size but it is reducing my prediction Accuracy score.

I'm using that model file inside the Flask and doing some real time predictions, Due to the large size i'm facing issues while predicting. So could anyone please help me on reducing the bert model size using pytorch and guide me on who to do the real time predictions.

$\endgroup$
2
  • $\begingroup$ Due to the large size i'm facing issues while predicting. Can you describe those issues? $\endgroup$
    – noe
    Jun 30, 2022 at 16:18
  • $\begingroup$ Thanks for the response, i faced issues like taking time for prediction, loading large size models in the flask, memory issues. $\endgroup$ Jul 1, 2022 at 13:41

1 Answer 1

1
$\begingroup$

Are you reusing an existing Bert model or are you training it from scratch?

In all the cases, you can apply several solutions to your model:

$\endgroup$
4
  • $\begingroup$ Thanks for the response, I'm using bert_model = BertModel.from_pretrained('bert-base-uncased') and from that written the classifier and trained with my own data set. $\endgroup$ Jul 1, 2022 at 13:44
  • $\begingroup$ Have you tried distill Bert? For information, if you train it from scratch only with your data, you can reduce its size considerably. $\endgroup$ Jul 1, 2022 at 14:15
  • $\begingroup$ huggingface.co/distilbert-base-uncased $\endgroup$ Jul 1, 2022 at 15:00
  • $\begingroup$ Note: I'm also developing a flask service with NLP models. I've written an article about it that might answers some questions about model optimization: medium.com/nerd-for-tech/… $\endgroup$ Jul 2, 2022 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.