This is somewhat of an open ended question and in some respects a literature request (I would love to be pointed to a survey paper if one exists).

Suppose I am constructing a neural network to make some arbitrary prediction (either categorical, or numeric, doesn't matter). With this network I am concerned primarily with speed of evaluation. Obviously, I want the network to give as accurate as possible predictions, but I'm more than willing to sacrifice some accuracy if it will make the network run faster (the evaluation faster, I don't care about training speed as long "reasonable"). What are some the techniques that might be employed to do this?

Broadly, I see two general approaches. First, you could build a network and then prune it. This could be done by removing neurons until the accuracy degrades past the point of acceptability. I could also add exit conditions at each, or some, hidden layers to skip the remaining layers if the result I have is "good enough". The second approach I see is to build a network from the start with speed in mind. I could do this by limiting the number of hidden layers and neurons. I could also use a simple activation function (such as linear or even identity) over the more common sigmoid.

What other techniques, if any, are there to increase the evaluation speed of a neural network? Also, if anyone is aware of any papers/articles/blogs/etc on this topic I would love to be pointed to them!

  • $\begingroup$ In addition to pruning your model, you could also look into using a mixed precision model or quantizing your model. This blog post from Roblox also shows some different steps to improve inference speed. $\endgroup$
    – Oxbowerce
    Nov 3, 2021 at 8:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.