I have a question (beginner :D) that is related to throughput and inference rate. Can the throughput and inference rate change as the model is trained or are the values for these parameters fixed? Many thanks to everyone who can contribute to the answer.
1 Answer
$\begingroup$
$\endgroup$
3
During the training of a model, either the parameters are calculated algorithmically or updated using Backpropagation (depending upon the algorithm and/or model architecture).
However, the architecture/algorithm and number of parameters do not change. As you would appreciate, that the inference time of a model depends on the architecture and number of parameters for a given hardware, training the model more number of iterations or on a larger training data would have no impact on the model's inference time throughput.
-
$\begingroup$ Thanks a lot for the answare my friend. Undestand. Sorry, just one more question. Could you suggest an algorithm to measure these two parameters using Google Colab? $\endgroup$– VernerNov 9, 2021 at 23:35
-
$\begingroup$ @Verner, I am not sure about the tools but I found this post informative towardsdatascience.com/…. In the end, it depends on the depth you want to get into - many a times I found it useful to just measure the inference throughput from a high level inference API like torchserve after some initial warmup. $\endgroup$– jdsuryaNov 10, 2021 at 14:24
-
$\begingroup$ Thank you @jdSuryaP for the suggestion. I'm to go read the link indicated. $\endgroup$– VernerNov 11, 2021 at 0:00