0
$\begingroup$

I've read some classics about comparison of ML Algorithms i.e.

Dietterich, T. G. (1997). Statistical Tests for Comparing Supervised Classication Learning Algorithms 1 Introduction. Science, 10(7), 1–24. Retrieved from http://dx.doi.org/10.1162/089976698300017197

However I feel totally lost about a specific problem.

Backgrond / Status Quo
I have two dataset ($N_1=552$, $N_2=543$) drawn from different populations.
Both contain the same set of features and the same criterion (7 class labels).
To simplify I will spare the details on preprocessing and hyperparameter tuning.
In the end I have two trained algorithms (i.e. two RandomForests: $RF_1$ & $RF_2$) for both datasets ($df_1$ & $df_2$) respectively

Goal / Aim
I want to know if it is better to train the algorithm using data drawn from population 1 and evaluate it using data drawn from population 2, or if the opposite is true. So which population generalizes better to the respective other.
To be more precise if a measure of the classification performance (i.e. Accuracy or Kappa) for the $RF_1$ (Random Forest trained in dataset 1) tested in $df_2$ is significantly higher (not caused by chance) than the performance for the $RF_2$ (Random Forest trained in dataset 2) tested in $df_1$.

$Acc(RF_1->df_2) > Acc(RF_2->df_1)$

Question
Is there an apropriate test for that? Is it as simple as the $\chi^2$ or a exact binominal test?

Edit
Or am I comparing apples and oranges, and there is no way one could compare this two classification results? I am very thankful for any direction you can give me.

Improve this question
$\endgroup$
7
  • $\begingroup$ You can't do that. To push your case towards extreme, imagine training one model on men and the other on Ravenous Bugblatter Beasts of Traal. Now you want to know how each of them works on the other population??? The whole idea of statistics and, by extension, machine learning, is to assume that your training sample is representative for the population. This is obviously not satisfied in your case. $\endgroup$
    – Igor F.
    May 12, 2020 at 7:23
  • $\begingroup$ Hi thank you for this really figurative advice (love the ref.). Sure such comparison wouldn't make sense and probably would result in two very low accuracies. But what about the case: comparing the typical student sample vs a sample from the general population (mean Age = 45)? In fact when using the second sample from the general pop. the model performs well in a separate test set as well as in the complete student sample. In contrast models trained with student data only perform well in a test set, but not so in the sample from the general population. $\endgroup$
    – Björn
    May 12, 2020 at 9:00
  • $\begingroup$ So you suggest there is no way one could reasonable compare these two accuracies? The two samples are representative of their respective population. And the basic concept of many test is to compare statistics from different population: ($\mu_1 - \mu_2$). However I dont have a mean (as I am obv. using set validation). $\endgroup$
    – Björn
    May 12, 2020 at 9:01
  • 1
    $\begingroup$ Regarding $\mu_1 - \mu_2$, you are actually asking whether the two samples are from the same population or not. Is this what you want to know about your two samples? $\endgroup$
    – Igor F.
    May 12, 2020 at 10:08
  • 1
    $\begingroup$ I believe you are trying to answer two distinct questions at once: 1) whether the algorithms perform the same and 2) whether two data sets are drawn from the same population. For the 1st question, you can take McNemar's test, but you need to asses the performance on one sample (maybe take the combined data sets?). For the 2nd question, I don't really know. Maybe try to train a new, separate classifier to distinguish the sets and see whether it performs better than a non-informative (null) classifier? $\endgroup$
    – Igor F.
    May 14, 2020 at 17:32

2 Answers 2

Reset to default
1
$\begingroup$

I understood it wrong ,here is the paper which discuss using multiple data set for the same classifier-

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.141.3142&rep=rep1&type=pdf

They conclude- " We theoretically and empirically analyzed three families of statistical tests that can be used for comparing two or more classifiers over multiple data sets: parametric tests (the paired t-test and ANOVA), non-parametric tests (the Wilcoxon and the Friedman test) and the non-parametric test that assumes no commensurability of the results (sign test). In the theoretical part, we specifically discussed the possible violations of the tests’ assumptions by a typical machine learning data. Based on the well known statistical properties of the tests and our knowledge of the machine learning data, we concluded that the non-parametric tests should be preferred over the parametric ones."

I recently learned about 5x2cv paired t test procedure to compare the performance of two models.

Please refer below-

http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/

It Implements the 5x2cv paired t test proposed by Dieterrich (1998) to compare the performance of two models.

Improve this answer
$\endgroup$
12
  • $\begingroup$ Thank you for your answer. This is a nice breakdown of the article I already read. Still I am puzzled because I have not one dataset but two, that are different: in that they are drawn from different populations (i.e. the one dataset is comprised of men and the other of women). $\endgroup$
    – Björn
    May 7, 2020 at 22:15
  • $\begingroup$ Well in your situation "I want to know if it is better to train the algorithm using data drawn from population 1 and evaluate it using data drawn from population 2, or if the opposite is true" this is essentially your null hypothesis and alternative one. all you have to do is assume a threshold and see with this test if your hypothesis is correct or not. This test gives you the p-value scores to these your problem exactly. no? $\endgroup$
    – BlackCurrant
    May 7, 2020 at 22:22
  • $\begingroup$ Oh in case your are wondering about its reference, i was taught this in advanced ML class so my professor vouches for it. if it helps at all. :-) $\endgroup$
    – BlackCurrant
    May 7, 2020 at 22:25
  • $\begingroup$ "this is essentially your null hypothesis and alternative one." Correct. I also know the treshhold (should be the usual $\alpha = .05$. What remains unclear to me is the type of test I should use for that. Is it just an exact binominal test to compare the accuracies $\endgroup$
    – Björn
    May 7, 2020 at 22:28
  • 1
    $\begingroup$ Okay so far I got that either t-test works for these problems or we compute the confidence interval of accuracy. $\endgroup$
    – BlackCurrant
    May 8, 2020 at 12:44
1
$\begingroup$

I believe it will be difficult to answer this question w/o knowing the underlying data.

Let's suppose,
N1 is from men's football and N2 is from women's football history
then both should be treated as separate data entity
or should be mixed to create train/test set if we have a compelling need.

What I will suggest -
Check the Mean, Max, Min, Variance, and Value counts(for categorical features) in both the set.

If both are not clearly distinct.
You can have the bigger one as Train and other as test

If they are e.g. One has All Germany and other has All France in "Country" feature,
Then both should be treated together as one data
Or you should have 2 models based on domain/business need

Concluding from the result w/o studying the Features can be deceiving as any observed pattern "can" be just by chance

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.