I suggest you to use an Autoencoder for dimensionality reduction. An Autoencoder is a Neural Network with a hourglass shape, that is meant to learn a compressed representation of your data. You can train it first on the data you already have, and then use it to extract a compressed representation at a time. In your case, what you need is an Autoencoder with Convolutional layers, since you need to process pixel data.
Once you trained it well enough you can generate a representation of each image from your stream one by one. You could load one at a time without memory issues.
Each time you generate a compressed representation, you can compare it with previous ones. Two pictures from the same person would end up being very similar to each other in the latent compressed space. Or you could even train a simple classifier that does the matching for you (that could be especially useful if you want to match two pictures of the same face, but turned in opposite directions for example).
The Internet and GitHub are plenty of Autoencoder works. Here I wrote a simple tutorial for a feedforward one in Python + TensorFlow 2. In your case, you need a Convolutional version of it. Your Encoder part will require 2D Conv layers, while the Decoder will require an inverse operation that can be done either with Upsampling layers or Transpose Conv layers (I've seen both implementations, this is an explanation of the two and how to use them).
An Autoencoder model in tensorflow.keras
would look like something like this:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
autoencoder = Sequential([
# Encoder
Conv2D(16, (3, 3), input_shape=(28,28,1) activation='relu', padding='same'),
MaxPooling2D((2, 2), padding='same'),
Conv2D(8, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2), padding='same'),
Conv2D(8, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2), padding='same'), # This is the compressed representation
# Decoder
Conv2D(8, (3, 3), activation='relu', padding='same'),
UpSampling2D((2, 2)),
Conv2D(8, (3, 3), activation='relu', padding='same'),
UpSampling2D((2, 2)),
Conv2D(16, (3, 3), activation='relu'),
UpSampling2D((2, 2)),
Conv2D(1, (3, 3), activation='sigmoid', padding='same')
])
This is a slightly modified version with Upsampling layers that I took from this Keras blog post. Alternatively, a version with Conv Traspose layers can be found here.
PS: Take a look at this article about finding alignments in hand written digits. Not exactly your problem, but rich of analogies IMHO.