Basic classification question

Question

I am wondering how I can manage a test data after using PCA or normalization and another thing like that in the classification because our model works on the representation given by its input vectors. For example, suppose you have used PCA in your training dataset to gain better accuracy or you have normalized (min-max) data. Now, you have developed a model and want to install it and label the new coming samples. You need to somehow apply PCA to each coming record and normalize that record. Applying PCA to one record will not yield the same effect of the PCA used in training phase and I think even it doesn't make sense to apply PCA to just one sample. So how can we manage these preprocessing techniques in the training phase in the test data, too?

Thanks in advance.

Answer 1

PCA is a matrix transformation from your original dataset to a set of orthogonal features. The transformation matrix which is applied to the training set is maintained and used in the future with your testing data such that the original testing set features will be mapped to same space as the training set transformed by PCA.

If the training set has $n$ instances and their are $m$ features, the training matrix is of size $n \times m$. The PCA transformation matrix is of dimensions $m \times k$, where $k$ is the number of retained PCA features, the top eigenvalues. Thus we can transform a single instance $1 \times m$, by the $m \times k$ transformation matrix. This results in a $1 \times k$ vector.

---

I have some text files that I vectorized using bag-of-words. The training set is shown below on the left side and the testing set is on the right side. Each row is a text file and the columns is the word count.

If we plot the first 2 features of this dataset we get

Now we will fit our PCA transformation matrix and we will apply this transformation to both the training and testing set.

from sklearn.decomposition import PCA
pca = PCA(n_components=2, copy=True)
pca.fit(X_train)
train_PCA = pca.transform(X_train)
test_PCA = pca.transform(X_test)

This gives the following plot. The purple and yellow points are the 2 different classes from the training set. Then the light blue points are from the testing set. You can see that the points in the testing set after being transformed by PCA will line up alongside the training set.