Now after certain preprocess we can repeatedly get two sets of data points, exhaustively enumerate one from the first set and one from the second set to form pairs. Each pair of these two data points have the same dimensionality, say two dimensional or one dimensional. The target is to measure the top n nearest ones among the distances of these pairs each time and perhaps finally the average of them. Euclidean distance is the simplest way to measure, but is there any more robust perhaps more complex way even applying neural networks to measure the distance between the pair? For instance, add a linear neural network layer after each data point, and measure the distance between the two outputs of the layer.

  • $\begingroup$ There are a lot of different standard metrics, and you could create your own which suits better your problem. In order to get an accurate answer, please consider providing more details. $\endgroup$ – Romain Reboulleau Dec 23 '19 at 6:47
  • $\begingroup$ Have edited. This time seems much more detailed. $\endgroup$ – piratesailor Dec 23 '19 at 8:31
  • $\begingroup$ Question is still quite broad, too broad in my opinion. $\endgroup$ – Romain Reboulleau Dec 23 '19 at 18:22
  • $\begingroup$ Please rewrite the question to make it more structured and readable. As of now, it's not clear enough IMHO $\endgroup$ – Leevo Dec 24 '19 at 10:31

It all depends how you frame your problem.

Whats complex? more parameters, more time complexity, more exotic looking etc...

Here are 2 cool ideas how to measure similarity:

Mahalanobis distance measure of the distance between a point P and a distribution D. Cool but more importantly its robust because you are using parameters of your distribution to measure the distance.

Neural networks. If we relax the definition of "measuring distance between data" then Check siamesse networks. The gist of it is, can you group similiar sequences of data together. Why I like it? cause of one-shot learning

  • 2
    $\begingroup$ If in the case the preprocess of two data points can't be trained, I mean, the preprocess is definitely given. Then what method can be adopted? Can we still use siamesse networks? $\endgroup$ – piratesailor Dec 23 '19 at 12:54
  • $\begingroup$ you can try. If it works great $\endgroup$ – vienna_kaggling Dec 23 '19 at 13:28
  • $\begingroup$ Would you please explain in this case, how is the network should be constructed? I mean the layers after the two data points etc. $\endgroup$ – piratesailor Dec 23 '19 at 14:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.