I am doing my final thesis in the field of Deepfakes and their detection. The final outcome is to have a binary classifier which could predict which video was updated and which was not. In other words, if video is fake, output number close to 1 and if video is real, output number close to 0. One of the reasons I started this project is because Facebook released a massive dataset called DFDC and they announced in media that the best solution got only 82% average precision on validation dataset and 65% average precision on the testing dataset. Source: https://ai.facebook.com/datasets/dfdc/

However, they also released a research paper describing the results: https://arxiv.org/pdf/2006.07397.pdf This is the part where I am confused. Research paper does not mention anything about 82% and 65%. Contradictory, in their research paper's figure you can see that Average Precision of the best score was around 0.98 (on the public testing dataset)

enter image description here

However, they also include the log loss scores and I managed to understand how Facebook calculates the values of 82% and 65% on public dataset and private dataset. What they do is described in the answer here: https://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss

which basically if the log-loss is 0.42, they take average precision as:

$$ AP = \frac{1}{e^{0.42}} $$

I was wondering if it is acceptable to use the above method because when I evaluate my score with scikit library and use: average_precision_score, I get much higher scores if to compare with the method which Facebook used as in https://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss

  • $\begingroup$ Wrong link maybe for the AP-LL connection? At least, I don't see such a claim there. $\endgroup$ – Ben Reiniger Feb 5 at 0:52
  • $\begingroup$ @BenReinigerI think it is correct link. In the answer of the link it says that L(p) = 0.5. This means that probability to the right class is 1/e^0.5 = 0.607. In the facebook announcement they say that average precision was 82.5% on public and 65.2% on private, but I can see log loss of the best public and private scores here: kaggle.com/c/deepfake-detection-challenge/leaderboard which are 0.192 and 0.428. So 1/e^0.192=82.5% and 1/e^0.428=65.2%, but what I am unsure about is if this metric is actually average precision. $\endgroup$ – MichiganMagician Feb 11 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.