Running two stage classification to predict relatively rare event?

Question

I have a very imbalanced sample in which I am trying to predict probability of a rare event (Out of around 25,000 observations, this event is observed around 30 times) and am reluctant to try over/under-sampling on that directly just because of the degree of imbalance.

Just to illustrate what I had in mind with an example: Let's say I'm trying to classify if a gem is an emerald.

But out of 25,000 samples, I only observe 30 emeralds. However, I have some other green stones like Jade and Peridot which bring up my observations to 300.

Would it be a good idea to determine P(Green Stone) and then P(Emerald | Green Stone) by running a two stage classification.

The latter stage will have imbalance ratio of 1 emerald : 9 non-emeralds, which might be more suitable for balancing.

Would appreciate any thoughts/insights on pursuing this idea.

Answer 1

Two-stage / hierarchical classification models are very useful.

Typically, the first stage is binary. It predicts the presence or absence of a low-rate event. If present, the second stage categories the type of event.

It is easier to train the models and typically the predictive ability of the models is higher.