Suppose that we have a dataset that some samples have the save value but with different target. It can be a regression issue or classification. What we should do with them? Should we remove them or that is normal and we can let these data be in training set?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
This is completely normal; leave them in. An easy example is in an ANOVA problem (which can be viewed as a regression) where multiple subjects in the same group (so same group "value" where group is the lone feature) will have different outcomes in $y$.
All this means is that, given your particular feature(s), you cannot get perfect predictions, but you should not expect to be able to get perfect predictions, anyway.
-
$\begingroup$ Yes, I was thinking about same flats but with different prices, however, that does not include enought features. Thanks $\endgroup$ Commented Jun 27, 2022 at 13:54
$\begingroup$
$\endgroup$
in addition to the answer from @Dave, consider that usually we start an assumption on the conditional distribution $P(y|x)$, for example for regression we assume Gaussian noise... this means that the target is something like $$y = f(x) + \epsilon \\ \epsilon \sim N(0, \sigma^2)$$ ... which means that is totally fine if you have multiple $x$ that maps to different $y$, and the model will find the MLE for that $x$ that explains better those 2 observations (even though this is unlikely in "normal"/"real world" datasets, since many times you have at least a continuous variable, which is unlikely to have 2 times the same value)
$\begingroup$
$\endgroup$
As others have explained for regression problems, leave them in.
For categorical, I think the situation is a bit more nuanced. It could be that you are actually in a multi-label setting, or it could be that the discrepancies are due to annotator disagreement. In the multi-label setting, you can merge labels, or consider re-annotating the data - giving annotators the option to choose multiple labels.
If you are convinced that you are actually in not in a multi-label setting then the conflicts are due to annotator disagreement - you have several options.
Do nothing, keep them in.
Drop data with conflicting labels in the training set.
Use soft-labels for training (score for each category is the probability that that is the correct category). There are several techniques for inferring the probabilities, the simplest of which is using the proportion of annotators that voted for the category. Another example includes weak supervision, where we take the annotators as labeling functions.
retrieve annotations that were used to calculate hard-labels, merge identical inputs and recalculate the "majority vote".
If you want to test some of these methods, just make sure you keep the conflicting data in the test set.
Personally, I would first access how often the issue is occurring, and if it much more prevalent in some classes. If it's infrequent enough I wouldn't do anything (keep them in).
