I've read some classics about comparison of ML Algorithms i.e.
Dietterich, T. G. (1997). Statistical Tests for Comparing Supervised Classication Learning Algorithms 1 Introduction. Science, 10(7), 1–24. Retrieved from http://dx.doi.org/10.1162/089976698300017197
However I feel totally lost about a specific problem.
Backgrond / Status Quo
I have two dataset ($N_1=552$, $N_2=543$) drawn from different populations.
Both contain the same set of features and the same criterion (7 class labels).
To simplify I will spare the details on preprocessing and hyperparameter tuning.
In the end I have two trained algorithms (i.e. two RandomForests: $RF_1$ & $RF_2$) for both datasets ($df_1$ & $df_2$) respectively
Goal / Aim
I want to know if it is better to train the algorithm using data drawn from population 1 and evaluate it using data drawn from population 2, or if the opposite is true. So which population generalizes better to the respective other.
To be more precise if a measure of the classification performance (i.e. Accuracy or Kappa) for the $RF_1$ (Random Forest trained in dataset 1) tested in $df_2$ is significantly higher (not caused by chance) than the performance for the $RF_2$ (Random Forest trained in dataset 2) tested in $df_1$.
$Acc(RF_1->df_2) > Acc(RF_2->df_1)$
Is there an apropriate test for that? Is it as simple as the $\chi^2$ or a exact binominal test?
Or am I comparing apples and oranges, and there is no way one could compare this two classification results? I am very thankful for any direction you can give me.