# How can I extract an optimized matrix of correlations from a larger data set?

Consider an Excel sheet containing a matrix of correlations between individual stocks and the combined portfolio as a whole:

How can I extract an optimized matrix such that most stocks have a low correlation to each other? If I simply go by the combined column, when I remove the ones that don't fit the criteria, I have a feeling it might not be optimal, as the combined column is affected by the ones that were removed.

## 1 Answer

Have not worked with stocks, but I have worked with feature selection. You could try to remove unwanted features (stocks in your case) one by one.

What I did for my tasks was to successively choose one feature, say $$X_k$$, out of the feature set $$\{X_j\}_{j=1\dots m}$$ and then build a linear regression model to predict k-th feature using remaining features:

$$\hat{X}_k = \alpha + \sum_{j \neq k} \beta_j X_j$$

You can then compute, e.g. Fraction of Variance Unexplained:

$$FVU_k=\frac{\left|X_k-\hat{X}_k\right|^2}{\left|X_k\right|^2}$$

High value tells you that current feature ($$X_k$$) cannot be predicted using other features, at least using a linear model - i.e. it is uncorrelated with them. Low value tells you that current feature is redundant at can be removed.

You can select the feature with lowest FVU, remove that, then recompute everything and remove next feature, and then keep going. This is a greedy algorithm, but, by virtue of one-by-one removal it will at least take into account multi-collinear relationship. In the past I would stop at $$FVU\ge 0.1$$.

At the end you will have a set of features (stocks) that are not correlated strongly. You could then rebuild the correlation matrix and recompute correlation of individual stocks with the combined <not actually sure what "combined" refers to>