I have two datasources A and B, and I want to check how several methods can affect the accuracy of my multi class models:
- If I use cross-validation with validate dataset to obtain the best hyper parameters.
- If I inject more data from source B.
- If I group some classes.
- If I use a different algorithm.
Lets assume I have four models (RF_1, RF_2, RF_3, XGB_4) that I want to compare with a first model (RF_0)...
Model | Description | Train | Validate | Test |
---|---|---|---|---|
RF_0 | Model with dataset A. Classes: C1, C2, C3. | 50%: A | 20%: A | 30%: A |
RF_1 | Model with dataset A and cross-validation. Classes: C1, C2, C3. | 50%: A | 20%: A | ?0 |
RF_2 | Model with datasets A and B. Classes: C1, C2, C3. | 50%: A+B | 20%: A+B | ?1 |
RF_3 | Model with dataset A, but some classes with similar characteristics are grouped. Classes: C1, C23 (C2+C3). | 50%: A | 20: A | ?2 |
XGB_4 | Model with dataset A, but different algorithm. Classes: C1, C2, C3. | 50%: A | 20%: A | ?0 |
Questions:
?0 - Does it need to be equal to RF_0 Test dataset to compare overall accuracy between models?
?1 - In this case, should I use the same Test dataset as RF_0? Or use a different dataset with 30% A+B. If I use a different Test dataset, can I still compare models?
?2 - Is it possible to compare with RF_0? Do I need to use Test dataset from RF_0 and group classes C2 with C3 inside this dataset? Or can I compare with new Test dataset?