# Approximation of long sequence of layers by one layer

Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of the model.

Layers from i up to j, effectively perform some nonlinear transformation: $$\mathbb{R}^{d_i} \rightarrow \mathbb{R}^{d_j}$$ Where $$d_i$$ and $$d_j$$ is the dimensionality of the input at $$i_{th}$$ and $$j_{th}$$ layers. Has there been any reseach on approximation of this nonlinear transform by some linear one - search of convolution layers with the activation function, which approximate the action of multiple layers as close as possible.

The loss function for this approximation can be simply an MSE error, where the inputs are the activations of $$\mathbb{R}^{d_i}$$ and outputs - activations $$\mathbb{R}^{d_j}$$. I see the possible problem, that mapping from one high dimensional space to another requires a lot od training data for reliable estimate, but, nevertheless I wonder, whether such an approach has been pursued in the literature?

First you create training data, by running your training data through the (trained) network, and recording the inputs to layer i, as your new training data (x) and the outputs from layer j as your desired answer (y).
You now simply (!) want to design a model that takes x and produces y. Ideally using fewer weights than layers i to j currently use.
Note: if your layers have residual connections, you will have to think carefully about how they handled. Taking the i-1th output and feeding it to the j+1th layer, when it previously expected the output of the jth layer, may go badly. Especially if the dimension of the data is different.