I am interesting in studying low footprint NN by replacing the activation functions with low-degree polynomial approximations. Doing this, I am fine with slight reduction in accuracy if I the training time reduces by orders of magnitude. I am aware of the issue of exploding/vanishing gradients on using polynomials as activation functions, but by making the input activations small by using some kind of normalization on the dataset, we can circumvent the problem. I have come across the following post on stackexchange discussion : https://stats.stackexchange.com/questions/299227/polynomial-approximations-of-nonlinearities-in-neural-networks which touches this subject briefly. Although it doesn't have anything concrete in it. I came across few papers like, https://openreview.net/forum?id=rkxsgkHKvH but this paper got rejected, so I am confident of their results. I am reaching out to the community for practises/papers in such a scenario.