Best way to deal with realistically imbalanced dataset for Regression problem

Question

I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data.

NOTE: The labels are continuous between 0 and 1 but binned for visual purposes.

These objects are from a simulation which accurately predicts the population numbers which we see in reality. (Hence the "realistically imbalanced" title)

Unfortunately, an object with label = 0 exhibits random looking motion and therefore the number of ways in which you can construct these objects tends to infinity..whereas those with label = 1 are highly ordered objects with not many variations in their structure and so have a limited number of ways in which they can appear.

To me this means that, in order to explore all possible variations of these objects in their domain would require a dataset who's "number of each class" would be the complete opposite as we see in the figure above. i.e. you need to observe exponentially more label ~ 0 objects than label = 1 objects in order to explore the objects' domain.

Obviously this is impossible and impractical. I'm curious as to what people think is the best method of approaching this issue?

I have thought about:

• Making this an object detection method whereby you discriminate between objects with label greater than or below a threshold such as 0.8. (I think this would be difficult given that the labels are continuous and so you would get muddying at the boundary).
• Joining two CNNs: one which models features that describe the label such as velocity data, and one which models properties that somehow link the level of physical randomness in the images to the corresponding labels.

I have already tried upsampling with a CNN classifier but with so few label ~ 0 images, the CNN quickly overfits and defaults to predicting them with a higher label.

Answer 1

You said that "an object with label = 0 exhibits random looking motion...whereas those with label = 1 are highly ordered objects". If you can see that, why should your model not be able to? Just train it and see if it works.

However, your dataset does not seem to be very large. You should think about transfer learning from some pre-trained model if your images/objects are not too strange.

There is no a priori need to handle an imbalance of labels, just be sure to compare your loss to a proper baseline, e.g. the label mean.

You said you have additional features, append them after flattening the convolutional layers. If the additional features are also 2D, just add them as a channel to your Input.

Answer 2

If you use logistic regression, you can try the following:

Oversample the minority class by passing class_weight="auto" to the LogisticRegression constructor. You may also want to set intercept_scaling=1e3 (or some other large value). See the docstring for details. Edit: As per sklearn version 0.17 'class_weight="balanced"'.

Change the intercept of the model. The class_weight should have made sure that you got the intercept (log-odds prior) for a 50/50 split, which can be turned into one for a 90/10 split by

prior = .9 lr.intercept_ += np.log(prior / (1 - prior)) - np.log((1 - prior) / prior) This mathematical trick is common in epidemiology (or so I've been told), where often you have a set of n_positive cases and a very small prior probability of disease, but getting a control group of actual size prior / (1 - prior) * n_positive is prohibitively expensive.

Similar tricks can be played with other probability models by multiplying the prior into their outputs, rather than folding it into the model directly. Naive Bayes (not a good probability model, but I'll mention it anyway) actually takes an optional class_prior argument.