I have two datasources A and B, and I want to check how several methods can affect the accuracy of my multi class models:
- If I use cross-validation with validate dataset to obtain the best hyper parameters.
- If I inject more data from source B.
- If I group some classes.
- If I use a different algorithm.
Lets assume I have four models (RF_1, RF_2, RF_3, XGB_4) that I want to compare with a first model (RF_0)...
|RF_0||Model with dataset A. Classes: C1, C2, C3.||50%: A||20%: A||30%: A|
|RF_1||Model with dataset A and cross-validation. Classes: C1, C2, C3.||50%: A||20%: A||?0|
|RF_2||Model with datasets A and B. Classes: C1, C2, C3.||50%: A+B||20%: A+B||?1|
|RF_3||Model with dataset A, but some classes with similar characteristics are grouped. Classes: C1, C23 (C2+C3).||50%: A||20: A||?2|
|XGB_4||Model with dataset A, but different algorithm. Classes: C1, C2, C3.||50%: A||20%: A||?0|
?0 - Does it need to be equal to RF_0 Test dataset to compare overall accuracy between models?
?1 - In this case, should I use the same Test dataset as RF_0? Or use a different dataset with 30% A+B. If I use a different Test dataset, can I still compare models?
?2 - Is it possible to compare with RF_0? Do I need to use Test dataset from RF_0 and group classes C2 with C3 inside this dataset? Or can I compare with new Test dataset?