# Preserving explained variance while reducing dimensionality

We have a function $f:R^N \rightarrow R$ and a set of points $D=\{ x\in R^N\}$. How is it possible to linearly lower the dimension of points to $M \ll N$ such that the fraction of explained variance* will be preserved as much as possible?

• we refer to the fraction of explained variance in prediction of output of $f$ using dataset $D$
• Have you tried principal component analysis? Jan 2 '17 at 10:20
– Dawny33
Jan 2 '17 at 15:21
• It felt too simple :) You know what? I'll make a more elaborate version and add that. Jan 3 '17 at 7:34
• @SvanBalen Please refer to my comment to his post Jan 11 '17 at 13:46
• en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma
– Emre
Jul 2 '17 at 19:35

$f$ is not defined for $\mathbb{R}^M$, so as well as reducing $x \in \mathbb{R}^N$ to $y \in \mathbb{R}^M$ you are creating a function approximator $g(y) \approx f(x)$.

I suggest using a neural network autoencoder with N dimension inputs and M dimension "bottleneck" layer. You will need to scale inputs and outputs. Using a standard auto-encoder, you will have to measure the fraction of preserved variance in $f(x)$ afterwards. The auto-encoder is not directly going to do that for you, instead it will, like PCA, attempt to use y to encode x (unlike PCA, it may do so nonlinearly).

You could take this further if $f(x)$ is differentiable. Instead of usual mean-squared error as the loss function, you could use $\frac{1}{k}\sum_{i=1}^{k} (f(x_{i}) - f(\hat{x_i}))^2$ - that will encourage the NN to preserve variance in $x$ that matters to $f(x)$. You will have to figure out the gradient function for this analytically. Also bear in mind I have not tried this, just attempted to match your requirement to some theory.

In addition, you will now have $g(y)$ as it will be $f(\hat{x})$. You can generate any $y$ from $x$ by running the first half of the autoencoder, and can generate any $\hat{x}$ from $y$ by running second half of the autoencoder.

You might also be able to adapt t-SNE by using your function $f(x)$ to generate distances that need to be preserved when reducing dimensions.

You could try to apply Principal Component Analysis, a form or ordination in which you linearly transform the vector space to maximize variance over the new dimensions. Consequently you'll end up with a new ordering in which more variance is contained in fewer dimensions, enabling you to consider less dimensions in your analyses.

Most, if not all, statistical software packages offer PCA as a component.

• Of course I'm aware of PCA. But it only applies if you want to predict the datapoints themselves not the output of function f. Immagine your datapoints are laid on an elipse, and function f is only varying along the short diameter. The first PC would be along the long diameter, while the output that I want is the short diameter (it's the dimension that best captures the changes in f) Jan 11 '17 at 13:45