Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
If I'm comparing performance between classification models on two different datasets should the number of samples per class, the number of classes, and features per sample be the relatively the same between the datasets?
Please let me know if you require any additional information.
From the comments, OP aims to compare two different kinds of biological data in terms of which one allows the best classification performance.
To compare anything reliably, it's always best to keep invariant as many variables as possible and vary only the main objects being compared. So ideally, this comparison would be made between the two different kinds of data for the same subjects, same number and distribution of classes. In my opinion the number of features per sample does not have to be identical, since it's related to the 'kind of data'.
If this ideal comparison is not possible (for example if the data is not obtainable), it's crucial at the very least to compare the two options on the same classification problem. This means that the classes (especially number and distribution) must be identical, or at least very close. Otherwise one of the problems might be easier to solve than the other, so there would be no way to be sure that the result of the comparison is due to the 'kind of data' being compared or the different setting of the problems.
