PCA formulation - Deep Learning book by Ian Goodfellow

Question

I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following:

we need to choose the encoding matrix D. To do so, we revisit the idea of minimizing the L2 distance between inputs and reconstructions. Since we will use the same matrix D to decode all the points, we can no longer consider the points in isolation. Instead, we must minimize the Frobenius norm of the matrix of errors computed over all dimensions and all points.

I can't understand the explanation for why can we no longer see the points in isolation. Would really appreciate if someone could help me out here.

Answer 1

He means that $D$ must work well on all data points, not just a single one. I.e., $$ z_i = Dx_i \;\;\text{ so that }\;\; || (I - D^TD)x_i ||_2^2 \;\;\text{ is small } $$ for all $i$. If we could choose a different matrix, say $M_i$ for each $x_i$, then we could minimize the error $E_i = || x_i - M_i^TM_i x_i ||_2^2$ separately for each $x_i$ in isolation. But since we want a single matrix that works for all points, so we stack them all together and instead optimize $$ D^* = \arg\min_D \mathcal{E}= \arg\min_D|| X - D^TDX||_F^2 $$ i.e., minimizing across the whole dataset simultaneously.

See e.g. here or here for more.