Suppose one is building a classifier that:
- takes as input the e-mail body text
- returns true or false if person X should be included in the address list
To build this classifier we have historical data of e-mail correspondence from a community of users Y.
- Persons John, Jill and Sally are members of Y
- Persons John and Jill are married
- Persons John, Jill and Sally are mutual friends
- Persons John and Sally are having an affair unbeknownst to Jill
The situation we want to avoid is:
- Every time Jill writes an intimate e-mail to John
- The classifier keeps recommending that she include Sally on the address list
So the question is how does one partition the training data to avoid such "contamination" of signals?