0
$\begingroup$

I have trained a DL image classification model. The model size is 250 MB. I want to expedite the inference timing for both GPU and CPU run. On searching a bit, the general trend for faster inference on GPU is using NVIDIA Tensor RT. While on CPU, it is to do pruning or quantization. I read a few articles about doing it on C++ but most of them are not succinct.
Can anyone please suggest the tried and tested way to solve the problem, I am a beginner and can't really find the mental framework to approach this problem. I can also save the model in ONNX and use TensorFlow framework, if they are faster and easier to use.
(Also if this is not a place ask questions on ML-Ops then please let me know the suitable forum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.