I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data.
NOTE: The labels are continuous between 0 and 1 but binned for visual purposes.
These objects are from a simulation which accurately predicts the population numbers which we see in reality. (Hence the "realistically imbalanced" title)
Unfortunately, an object with label = 0 exhibits random looking motion and therefore the number of ways in which you can construct these objects tends to infinity..whereas those with label = 1 are highly ordered objects with not many variations in their structure and so have a limited number of ways in which they can appear.
To me this means that, in order to explore all possible variations of these objects in their domain would require a dataset who's "number of each class" would be the complete opposite as we see in the figure above. i.e. you need to observe exponentially more label ~ 0 objects than label = 1 objects in order to explore the objects' domain.
Obviously this is impossible and impractical. I'm curious as to what people think is the best method of approaching this issue?
I have thought about:
- Making this an object detection method whereby you discriminate between objects with label greater than or below a threshold such as 0.8. (I think this would be difficult given that the labels are continuous and so you would get muddying at the boundary).
- Joining two CNNs: one which models features that describe the label such as velocity data, and one which models properties that somehow link the level of physical randomness in the images to the corresponding labels.
I have already tried upsampling with a CNN classifier but with so few label ~ 0 images, the CNN quickly overfits and defaults to predicting them with a higher label.