3
$\begingroup$

In the context of face recognition I have the following histogram:

enter image description here

blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the distances for cross matches (different persons).

The distance is the value returned by the neural networks as the result of a comparison between two faces, how much two "faces vectors" (embeddings) differ.

I'm looking for a function that, given a distance, can tell how likely it is for two pictures to be from the same person.

This function should be look like this red line (with a different y-scale):

enter image description here

So with distance 0.5 it is extremely likely, with distance 1.4 it is close to a 50% chance.

Is there such a function? How is it called?

It is similar to this question but also very different. In my case a very small z-value (distance) still means high confidence even if it is many standard deviations away from the mean. The same is true for a value like 1.0.

This question is very similar too but I'd like to extract the distance to probability mapping function from actual measurements, on a reference dataset, and not from the distance by itself. So this should not depend on the loss or distance definition but on the data distribution alone.

Improve this question
$\endgroup$
5
  • $\begingroup$ what is the red line? $\endgroup$
    – Leevo
    Jan 17, 2020 at 18:38
  • $\begingroup$ @Leevo The red line is the intuitive shape of the function I have in mind $\endgroup$
    – lorenzo
    Jan 17, 2020 at 23:21
  • $\begingroup$ What did you come up with at end? I am facing a similar problem at the moment! $\endgroup$
    – Romanzo Criminale
    May 8, 2020 at 10:43
  • $\begingroup$ @RomanzoCriminale Not much, you can play with integrals or integral ratios but nothing I really liked. Also, if you want to know the real world probability for an access with score x to be legit you also need to take into account how likely an illegal access attempt is in the first place. $\endgroup$
    – lorenzo
    May 13, 2020 at 17:25
  • $\begingroup$ Thanks. I'm using the integral ratios as well. I don't think there is a perfect solution anyway. Not sure what you meant exactly with your last sentence. $\endgroup$
    – Romanzo Criminale
    May 14, 2020 at 9:58

1 Answer 1

Reset to default
3
$\begingroup$

If I had to calculate such a function, I would:

Calculate the probability (not Z-score) for $x$ in a two-tailed test (https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) for the probability that $x$ belongs to the distribution for both distributions. Then you have $p_s(x)$ and $p_c(x)$ for self-match/cross-match. Don't use probability $0.1\%$ to reject, but to compare.

Then calculate $$\frac{p_s(x)}{p_s(x) + p_c(x)}$$

Which tends to $1$ if $p_s(x)$ is the larger term, is $0.5$ if they are the same, and tends to $0$ if $p_c(x)$ is the larger term.

At the far tails, $p_s(x)$ and $p_c(x)$ both tend to zero, but the order of magnitude will select the good probability.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I'm missing the reference to "probability 0.1%" (p-value?). The Ps for 0.1 and for 0.9 should be different even if both are at the same distance from the mean, I'm not sure this if is accounted for. Also, considering that Pc is practically zero up to 1.2 this formula gives 1 for the whole [0, 1.2] range: intuitively is not ideal (even if it might be correct) as I would expect it to decrease lightly while the distance increases. $\endgroup$
    – lorenzo
    Jan 18, 2020 at 10:04
  • $\begingroup$ The probability that 1.4 belongs to either one of the distributions is very small, I just picked $0.1\%$. Typically you'd reject that the point belongs to the distribution, but given that you need to select a distribution, you still use it in your formula. Your intuition applies to distributions that have significant overlap. These don't and you can expect a steep descent and a clear decision boundary. $\endgroup$
    – Pieter21
    Jan 18, 2020 at 10:29
  • $\begingroup$ This works, but what bothers me is that the idea of distance is completely lost. This would be identical if the bells represent the sizes of dogs of two different breeds. Here 0.1 and 0.9 means very different things even if 0.1/0.1 and 0.9/0.9 gives the same result. Assuming that the strict probability is correct in this way, then maybe I'm looking for a different kind of confidence metric specific for distance based comparisons. $\endgroup$
    – lorenzo
    Jan 19, 2020 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.