1
$\begingroup$

I have a question (beginner :D) that is related to throughput and inference rate. Can the throughput and inference rate change as the model is trained or are the values for these parameters fixed? Many thanks to everyone who can contribute to the answer.

$\endgroup$

1 Answer 1

1
$\begingroup$

During the training of a model, either the parameters are calculated algorithmically or updated using Backpropagation (depending upon the algorithm and/or model architecture).

However, the architecture/algorithm and number of parameters do not change. As you would appreciate, that the inference time of a model depends on the architecture and number of parameters for a given hardware, training the model more number of iterations or on a larger training data would have no impact on the model's inference time throughput.

$\endgroup$
3
  • $\begingroup$ Thanks a lot for the answare my friend. Undestand. Sorry, just one more question. Could you suggest an algorithm to measure these two parameters using Google Colab? $\endgroup$
    – Verner
    Nov 9, 2021 at 23:35
  • $\begingroup$ @Verner, I am not sure about the tools but I found this post informative towardsdatascience.com/…. In the end, it depends on the depth you want to get into - many a times I found it useful to just measure the inference throughput from a high level inference API like torchserve after some initial warmup. $\endgroup$
    – jdsurya
    Nov 10, 2021 at 14:24
  • $\begingroup$ Thank you @jdSuryaP for the suggestion. I'm to go read the link indicated. $\endgroup$
    – Verner
    Nov 11, 2021 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.