# How can I apply PCA to KNN?

I want to know the do not want to how to use library

I will denote a $n\times p$ data matrix by $X$, where $n<p$. That is, each row of $X$ is one sample data with $p$ feature variables.

By using singular vector decomposition method, I can decompose $X$ into $A$, $B$, and $C$ such that $X=ABC$, where $A$ is a $n\times n$ matrix satisfying $A^TA=I_{[n\times n]}$, $B$ is a $n\times p$ matrix whose diagonal elements are all non-negative and real but non-diagonal elements are all zeros, and $C$ is a $p\times p$ matrix such that $P^TP = I_{[p\times p]}$. Without the loss of generality, I assume that $B$ is obtained satisfying that the diagonal elements of $B$ is descending order, i.e., $b_{i,i} > b_{j,j}$ for $i<j$.

The score matrix, $S$, is obtained by $S=XC$, whose dimension is $n\times p$.

In order to reduce the dimension of variables, we do as follows: Denoting the $i$-th column of $C$ by $c_i$, which is a loading vector with respect to $i$-th principal component. Hence, deleting $c_{k+1}$ to $c_{p}$, I can obtain $C_k = [c_1~c_2~\cdots~c_k]$. The dimension-reduced score matrix, $S$, is obtained by $S_k=XC_k$.

In this situation, how can I run KNN? Actually, I already made my own KNN code, whose input is trainX, trainY, and testX, and whose output is evaluatedY. I want to compare (i) basic KNN and (ii) dimension-reduction-applied KNN.

I obtained error rate of (i) basic KNN using 5-fold cross validation method.

yhat = myKNN(trainX, trainY, testX);
errRate = myErr(y, yhat);


To obtain error rate of (ii) dimension-reduction-applied KNN, is the following correct?

trainS= myDimensionReduction(trainX, k)
testS= myDimensionReduction(testX, k)
yhat = myKNN(trainS, trainY, testS);
errRate = myErr(y, yhat);


where myDimensionReduction is a function that converts a data matrix $(n\times p)$ into a reduced data matrix $(n\times k)$.

Everything looks good. However, what I doubt about is the myDimensionReduction function. When you apply the PCA on the test data, you have to multiply $X_{test}$ by the matrix $C$ that has been generated from the training data, not by the test data. This is to stress that the test data should not be used in any step of the training process. If this is what the myDimensionReduction function does, that is alright. Otherwise, if this function creates the matrix $C$, then it is not right.
I know you didn't ask for the library, but I think it makes sense in here: The whole philosophy of scikit-learn is to create objects that correspond to algorithms, use the training data with the fit method and apply the transform method to the test data.
• Before using PCA, when testing a vector $x_q$ whose size is $1\times p$, I considered the $K$ nearest neighbors are $x_i$'s with the smallest $\lVert x_i-x_q\rVert$. However, when using PCA, when testing $x_q$, is it fine to consider the $K$ nearest neighbors as $K$ data $x_i$'s with the smallest $\lVert x_i C_k - x_q C_k\rVert$ ? – Danny_Kim Jun 6 '18 at 17:00
• Sorry to keep asking, denoting by $c_{i,j}$ the $i$-th row and $j$-th column of $C_k$, is it reasonable to estimate the contribution of a variable $x_i$ by $\sqrt{\sum_{j=1}^{k}c_{i,j}^2}$. – Danny_Kim Jun 6 '18 at 17:05