# How to run PCA and KNN on big-data

I work with python and images of watches (examples: watch_1, watch_2, watch_3). My aim is to take a photo of a random watch and then find the most similar watches to it in my database. Obviously, one main feature which distinguishes the watches are their shape (square, rectangular, round, oval) but there are also other ones.

For now, I am just running a PCA and a KNN on rgb images of watches to find the most similar ones among them. My source code is the following:

import cv2
import numpy as np
import os
from glob import glob
from sklearn.decomposition import PCA
from sklearn import neighbors
from sklearn import preprocessing

data = []

# Read images from file
for filename in glob('Watches/*.jpg'):

height, width = img.shape[:2]
img = np.array(img)

# Check that all my images are of the same resolution
if height == 529 and width == 940:

# Reshape each image so that it is stored in one line
img = np.concatenate(img, axis=0)
img = np.concatenate(img, axis=0)
data.append(img)

# Normalise data
data = np.array(data)
Norm = preprocessing.Normalizer()
Norm.fit(data)
data = Norm.transform(data)

# PCA model
pca = PCA(0.95)
pca.fit(data)
data = pca.transform(data)

# K-Nearest neighbours
knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)
distances, indices = knn.kneighbors(data)

print(indices)


However when I try to run this script for more than 1500 rgb images then I get a MemoryError at the point where the data are processed by the PCA method.

Is this normal for a pc with 24GB RAM and 3.6GHz Intel Core CPU without any discrete GPU?

How can I overcome this?

Shall I use another method like Incremental PCA (or a deep learning algorithm) or simply shall I buy a discrete GPU?

• Where are you getting the memory error? Is it when you are storing the images into the list data? – kingledion Feb 20 '18 at 15:52
• Thank you for your useful note. I edited my post to answer it. I get it at the point where the data are processed by the PCA. – Outcast Feb 20 '18 at 15:56