Multi Output Classification - Outputs not making sense relative to one another

Question

Let's say I have a Multi Output Binary Classification Problem, but where the classes are related. i.e if one class = 1, then the other must = 1.

The standard is to have 2 output layers, each with 1 output unit each and using sigmoid activation function.

As an example, let's say I am trying to predict if I will cross the 10km & 15km mark during a marathon.

How would I ensure that the network knows that if I've crossed the 15km mark, then I have also crossed the 10km mark? How would I make the probabilities consistent? P(Cross 10km) >= P(Cross 15km).

Should the network naturally learn from the dataset (i.e it is not possible to cross the 15km mark if the 10km hasn't been crossed)?

Answer 1

The network won't always make predictions consistent with your known constraint.

The jargon here is "ordinal regression" (the term "ordinal classification" seems more apt, but isn't often used). See Wikipedia or our tag ordinal-regression; in particular Cost function for Ordinal Regression using neural networks asks about neural networks.

In this stats.SE answer, a method is proposed to enforce that constraint: the output neurons have the same weights from the previous layer, and only different biases are learned.

One last comment on your example: if you have access to the actual distance ran, perhaps you can leverage the additional information to perform a regression, and layer on some distribution assumption/learning to get probabilities out from a predicted expected value.