I used torch.quantization.quantize_dynamic
to reduce the model size but it is reducing my prediction Accuracy score.
I'm using that model file inside the Flask and doing some real time predictions, Due to the large size i'm facing issues while predicting. So could anyone please help me on reducing the bert model size using pytorch and guide me on who to do the real time predictions.
Due to the large size i'm facing issues while predicting
. Can you describe those issues? $\endgroup$