# Using pretrained LSTM and Bert Models in CPU Only Environment - How to speed up Predictions?

I have trained two text classification models using GPU on Azure. The models are the following

1. Bert (ktrain)
2. Lstm Word2Vec (tensorflow)

Exaples of the code can be found here: nlp

I saved the models into files (.h5) for later use. The files are big e.g. 27,613kb for the lstm and 1.2 gb for bert.

I loaded the models and in a computer where only CPU is available. They both work fine but the model.predict(text) function is super slow predicting the class of the text e.g. on average 1 tweet sized message per second.

Adding GPU on the computer is not an option. I wonder if there is another way to make it run faster? e.g. train the models in a different way (without compromising accuracy) or save the model in a different file format?

• I don't think there is any way to speed up the actual prediction time of the exact same model. The fastest way I can think of is loading the model once, and then predicting in batches. Do you do with torch.no_grad() and set model.eval()? – N. Kiefer Oct 15 '20 at 9:52
• @N.Kiefer the question specifies that the models are for tensorflow (ktrain is a tf.keras wrapper). – ncasas Oct 15 '20 at 13:32