1
$\begingroup$

Suppose I want to predict some event probability with a set of features. Some of the features may be gender or some other categorical variable. Assume the probabilities are well calibrated.

Now, say, I need to report the average predicted probability for males and females (and maybe some other subset of features).

Is there a difference between

  1. averaging the probabilities of the full model or
  2. simply fitting a model on only the subset of variables?

And if so why? The full model would outperform the simpler model by a large margin. An example would also be appreciated.

$\endgroup$
3
  • 1
    $\begingroup$ Could you please clarify (1) and (2) with an example like $P(event|\text{male})$ vs ....? $\endgroup$ – Esmailian Mar 22 '19 at 18:04
  • 1
    $\begingroup$ For example $P(event|x_1)$ or $AVG_{x_n:(x_{n,1})}P(event|x_{n,1})$ vs averaging over all possible feature values $AVG_{x_2} P(e|x_1, x_2)$ or all data points $AVG_{x_n:(x_{n,1}, x_{n,2})}P(e|x_{n,1}, x_{n,2})$ ? $\endgroup$ – Esmailian Mar 22 '19 at 18:25
  • $\begingroup$ both are basically a group by male/female + avg(prob) but in 1. I fit the model on other features that are not in the "group by" whereas 2. considers just features that I'll be grouping on $\endgroup$ – oW_ Mar 22 '19 at 18:27
2
$\begingroup$

A very thought provoking question.

Surprisingly, second approach (subset) is better in theory. The first one (full) is an unbiased estimator of second one (subset). That is, expectation of average of probabilities in full model is equal to the probability in subset model.

I said "in theory", because we assume that full and subset models estimate the true P(event | full) and P(event | subset) exactly respectively (zero error, perfect generalization, etc.). In practice, choice of learning algorithm and training set affects how accurate they can fulfill this assumption. A model could be Naive Bayes, Logistic Regression, Neural Network with a SoftMax layer, etc.

Proof

Let

  1. $F_f$ denote the full features,
  2. $F_s$ denote the subset features (that we group on),
  3. $F_d=F_f-F_s$ denote the features that we average out,
  4. $\boldsymbol{A}=\{x|F_s(x)=a\}$ denote the set of all possible instances with subset features equal to $a$, and
  5. $A \subset \boldsymbol{A}$ denote the observed subset of $\boldsymbol{A}$ in training set.

Here is the average of probabilities in full model (first approach): $$\begin{align*} P_f(e|F_s=a)&=\frac{1}{N_A}\sum_{x\in A}P(e|F_d(x),a) \end{align*}$$

Expectation of above average is: $$\begin{align*} E[P_f(e|F_s=a)]&=\frac{1}{N_A}\sum_{x\in A}E[P(e|F_d(x),a)]\\ &=\frac{1}{N_A} N_A E[P(e|F_f(X))|X \in \boldsymbol{A}]\\ &=\sum_{b}P(e|F_d=b,F_s=a)P(F_d=b|F_s=a)\\ &=\sum_{b}P(e,F_d=b|F_s=a)\\ &=P(e|F_s=a) \end{align*}$$

where $P(e|F_s=a)$ is the output of second approach.

Example

To verify, here is an example:

$e=\text{has_job}$

$F_f=\{\text{sex}, \text{has_degree}\}$

$F_s=\{\text{sex}\}$

$F_d=\{\text{has_degree}\}$

Full model sees:

x     sex  has_degree  has_job
1     f    1           0
2     f    1           1
3     f    0           0

4     m    0           0
5     m    0           0
6     m    1           1

And builds

sex  has_degree  P(has_job=1|sex, has_degree)
f    0           0
f    1           0.5
m    0           0
m    1           1.0

And answers $P(\text{has_job}=1|\text{sex}=f)$ with

$$\frac{1}{3}\sum_{x \in \{1, 2, 3\}}P(\text{has_job}=1|x)=(0.5+0.5+0)/3=0.33,$$

and answers $P(\text{has_job}=1|\text{sex}=m)$ with

$$\frac{1}{3}\sum_{x \in \{4, 5, 6\}}P(\text{has_job}=1|x)=(0+0+1.0)/3=0.33$$

On the other hand, subset model sees:

x     sex  has_job
1     f    0
2     f    1
3     f    0

4     m    0
5     m    0
6     m    1

And builds

sex  P(has_job=1|sex)
f    0.33
m    0.33

And answers $P(\text{has_job}=1|\text{sex}=f)$ with 0.33, and $P(\text{has_job}=1|\text{sex}=m)$ with 0.33.

$\endgroup$
4
  • $\begingroup$ Thank you for the answer! I understand that the two are the same when you look at counts of the events. They will be the same no matter in what order you perform the aggregation. I'm talking about probabilities that stem from a machine learning model (say some neural net etc.). $\endgroup$ – oW_ Mar 22 '19 at 23:11
  • 1
    $\begingroup$ @oW_ My pleasure, A neural network is trying to estimate P(e|~), so the final result would be the same. The shortcoming of NN to model P(e|~) is error. I think, there is not much to say about the error either mathematically or by example. $\endgroup$ – Esmailian Mar 22 '19 at 23:19
  • $\begingroup$ You seem to argue that the correct probabilities are based on counts, whereas a model would simply approximate them. Then there would be no need for machine learning at all. A ML model provides better generalization and should provide more robust probabilities (e.g. in the case where you have many sparse categories because it can borrow strength from related categories). My question is whether or not on average they are also more accurate, or if that higher accuracy of the full model over the subset model gets lost when aggregating. Hope this helps. $\endgroup$ – oW_ Mar 22 '19 at 23:35
  • 1
    $\begingroup$ @oW_ I think generalization power of an ML model is relevant to "how accurate model estimates P(e|~)?", meaning: better generalization, better estimation, less error. We can compare the two approach by assuming the perfect generalization (i.e. no error), otherwise we cannot arrive at a conclusion, or give an example. $\endgroup$ – Esmailian Mar 22 '19 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.