I am a grad student doing research using generative machine learning with pytorch, and I have generated a set of points. I would like to check how similar these new points are to the points I used in my training data, using nearest neighbor distance to see if the new points are actually new. My first thought was to just use torch.cdist to get a matrix of Euclidean distances and then take the minimum column-wise to get the smallest distance for each point in the new generated data.

The problem is that my training data set is around 7 million points, which seems to be causing issues when I try to use the method I described above (since there are about 10 thousand new points to check against this means my cdist matrix would have something like 70 billion entries, which I am guessing is using a lot of RAM?)

Is there an easier, time-efficient way to calculate these values for such a large dataset? Or should I look into other methods to check how 'unique' my new points are?

  • $\begingroup$ Are you obliged to train your data over 7 million points? Generally speaking, we take a random sample of a few thousands, we see the result, improve the model and then increase the amount. Random data is quite representative of the whole dataset, but it depends on the project. $\endgroup$ Sep 3 at 13:52
  • $\begingroup$ Not obliged to, but I have already trained the network and generated these points which have other desirable properties, so I would like to see if they are truly unique or if they end up being too close to my training set data $\endgroup$ Sep 3 at 21:24
  • $\begingroup$ Maybee using another data structure would be more suitable for this amount of data ? a hashatble with 'distance' -> list of points ? and/or just keeping the smaller distance values if it is what your are looking for ? or a table (point) -> (nearest other point) $\endgroup$
    – Malo
    Sep 4 at 13:48
  • $\begingroup$ Update: I was able to get it to work by just looping for each entry in the new generated points, although it does take a while to run $\endgroup$ Sep 7 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.