I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in my training data, using nearest neighbor distance to see if the new points are actually new. My first thought was to just use torch.cdist to get a matrix of Euclidean distances and then take the minimum column-wise to get the smallest distance for each point in the new generated data.
The problem is that my training data set is around 7 million points, which seems to be causing issues when I try to use the method I described above (since there are about 10 thousand new points to check against this means my cdist matrix would have something like 70 billion entries, which I am guessing is using a lot of RAM?)
Is there an easier, time-efficient way to calculate these values for such a large dataset? Or should I look into other methods to check how 'unique' my new points are?