For image classification tasks, is there a practical difference between using weighted loss functions vs. using weighted sampling? (I would appreciate theoretical arguments, experience or published papers, anything really.)
Some details:
By "weighted sampling", I mean attributing different sampling probabilities for each sample in the training set.
By "weighted loss functions", I mean weighting error terms differently depending on the sample considered.
First off, I would point out that both concepts can very well coexist. Let's take the following example:
Image classification including 2 classes and samples covering 2 domains. The classes are imbalanced and one domain is "harder" than the other. You could use weighted sampling to sample more examples from the "harder" domain while simultaneously use a weighted loss to counter the class imbalance.
Now let's look at some pros and cons:
Weighing each sample implies that you have knowledge of all samples and score them all. From a practical perspective, this is not always possible or feasible. Indeed, in cases where training samples are streamed, you don't control which samples come your way and weighted sampling is impossible. Or in cases where you continually update the weights of each samples.
On the other hand, it is much more practical to update the weights of your samples than to modify your loss throughout training.
On the plus side, a weighted loss isn't dependent on how the instances are sampled, which can be more practical. On the down side, if you sample your data in such a way that most instances have low weights, your model will not update rapidly.
