0
$\begingroup$

I have two datasources A and B, and I want to check how several methods can affect the accuracy of my multi class models:

  1. If I use cross-validation with validate dataset to obtain the best hyper parameters.
  2. If I inject more data from source B.
  3. If I group some classes.
  4. If I use a different algorithm.

Lets assume I have four models (RF_1, RF_2, RF_3, XGB_4) that I want to compare with a first model (RF_0)...

Model Description Train Validate Test
RF_0 Model with dataset A. Classes: C1, C2, C3. 50%: A 20%: A 30%: A
RF_1 Model with dataset A and cross-validation. Classes: C1, C2, C3. 50%: A 20%: A ?0
RF_2 Model with datasets A and B. Classes: C1, C2, C3. 50%: A+B 20%: A+B ?1
RF_3 Model with dataset A, but some classes with similar characteristics are grouped. Classes: C1, C23 (C2+C3). 50%: A 20: A ?2
XGB_4 Model with dataset A, but different algorithm. Classes: C1, C2, C3. 50%: A 20%: A ?0

Questions:

?0 - Does it need to be equal to RF_0 Test dataset to compare overall accuracy between models?

?1 - In this case, should I use the same Test dataset as RF_0? Or use a different dataset with 30% A+B. If I use a different Test dataset, can I still compare models?

?2 - Is it possible to compare with RF_0? Do I need to use Test dataset from RF_0 and group classes C2 with C3 inside this dataset? Or can I compare with new Test dataset?

Improve this question
$\endgroup$
1
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Jan 26 at 5:25

1 Answer 1

Reset to default
1
$\begingroup$

In a supervised setting, the problem is to model the following joint distribution:

$$\begin{array}{lcl}P(X,Y) & = & P(X) \cdot P(Y \mid X) \\ & = & P(Y) \cdot P(X \mid Y)\end{array}$$

where $X$ is the input space and $Y$ the output space.

  • Yes, because the comparison is fair. Indeed, RF_1 is the cross-validated version of RF_0 and XGB_4 is another modal trained in the same scenario.
  • No, you must consider data distribution shifts. For example, $P(X)$ changes (since you add more data), but $P(Y \mid X)$ remains the same, also called covariate shift.
  • No, for the same reason as above. For example, $P(X)$ remains the same, but $P(Y \mid X)$ changes, called concept shift, or $P(Y)$ changes, but $P(X \mid Y)$ remains the same, called label shift.

Therefore, the latter two examples are like comparing apples with pears.

If you are interested in the (challenging) topic of data distribution shifts, you can read this chapter by Chip Huyen.

Improve this answer
$\endgroup$
3
  • $\begingroup$ But in the second example (?1), if I add more data of the rare class to train RF_2, I expect to obtain better metrics when comparing with the same Test data from RF_0, right? I am having an hard time understanding this phrase "However, for an example with a given age, such as above 40, the probability that this example has breast cancer is constant. So P(Y|X), the probability of having breast cancer given age over 40, is the same." How can be the same if I increased my training data? $\endgroup$
    – Just_4n0th3r_Pr0gr4mm3r
    Feb 7 at 11:30
  • $\begingroup$ From a philosophical point of view, imagine two people, $P_1$ and $P_2$, trained to pass an exam at the University. Assume that $P_1$ had less training than $P_2$ for personal reasons (e.g., had a part-time job). The exam must be the same to asses who is better than who. Nevertheless, it will be an unfair comparison because $P_2$ will likely get a better score (for his higher class attendance). What I mean with this example is that the comparison must be fair in input and output; a model with more training data will likely perform better (though this is no mathematical proof). $\endgroup$
    – Eduard
    Feb 7 at 13:16
  • $\begingroup$ Now for your second question. The phrase is preceded by "You might have more women over the age of 40 in your training data than in your inference data, so the input distributions differ for your training and inference data". Therefore, the example follows. $\endgroup$
    – Eduard
    Feb 7 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.