2
$\begingroup$

In papers, a problem have multiple ways of solving it. E.g. For an imbalance dataset, there are method A, B, C fixes during modelling. However, how can I objectively evaluate which one to use, given these fixes are proposed by different papers, each of them have completely different problems in domain (e.g. some of them are computer vision related, some of them text related), and their evaluation metric or loss function differs completely?

How can I judge which method to adopt for solving the problem of 'class imbalance', for instance, when all the settings across these fixes are just different, which makes it impossible to compare?

I'm trying to see if there's a standardized way to pick the method without implementing every single A,B,C,D,...,Z, as in industry we don't have that much time reimplementing every single methods.

Thanks in advance.

$\endgroup$

1 Answer 1

1
$\begingroup$

This is a good question. Unfortunately, there is no 100% satisfying answer for you.

The Problems

Let's start with the problems, you are facing:

No-Free-Lunch

Have you ever heard about the no-free-lunch-theorem? In short: there is no method that is always the best. This does not mean that you cannot give recommendations for algorithms, but you can never be 100% sure. Your use case might be the one in a million example where an algorithm does not perform well.

Results in research papers

Unfortunately, results in research papers are often not very well comparable. There are many reasons (some of them named by you):

  • Different use cases: Often these papers are driven by slightly different use cases and solve slightly different problems. Although many algorithms can be adopted to other use cases, this is not part of the paper or the evaluation.
  • Different datasets / domains: Often papers are evaluated on different datasets. Researchers tend to use datasets they have at hand. Although for some problems, there are datasets that everyone uses, for many others there is not standard.
  • Different ways of evaluation: Even if the use case and the datasets are the same, the evaluation / experiments might be different. It starts with the metrics, e.g. some use accuracy, other use roc-auc. But also some papers might do a preprocessing or some previous filtering of samples.

But there are also some critical reasons around the scientific process. Summarized under the term publish or perish there are factors that increase the need to get papers published. So if a method is not better in terms of roc-auc, why not use accuracy instead? You can see that such factors can drive researchers to make there papers less comparable among each others.

Some years ago, there where some notable papers such as [1,2] that showed that there was no real improvement over a decade in information retrieval. Just the base line methods in each paper where "wisely" chosen to report improvement over the baseline.

What con one do?

So, if the results reported in scientific papers are hard to compare, what are your options?

Papers with code

Nowadays, many papers come with source code. This makes it relatively easy to download the code and apply it on your own data. This should give you a much better comparison for your specific use case.

Learn from others

Publishing a new method is just the first step. People start to use these methods and many do so openly. So look at kaggle compations related to your use case (or create one on your own). Which methods win there?
Look a blogs, such as towards data science and see what people report from the real-world application.

References

[1] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Has adhoc retrieval improved since 1994?
[2] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don't add up: ad-hoc retrieval results since 1998

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.