I have trained a DL image classification model. The model size is 250 MB. I want to expedite the inference timing for both GPU and CPU run. On searching a bit, the general trend for faster inference on GPU is using NVIDIA Tensor RT. While on CPU, it is to do pruning or quantization. I read a few articles about doing it on C++ but most of them are not succinct.
Can anyone please suggest the tried and tested way to solve the problem, I am a beginner and can't really find the mental framework to approach this problem. I can also save the model in ONNX and use TensorFlow framework, if they are faster and easier to use.
(Also if this is not a place ask questions on ML-Ops then please let me know the suitable forum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.