# Sequence multi-class classification only learns a few outputs

I have a multi-event delineation problem, where given a signal, I have an output with the same signal length. Something like 0011002200, where each unique number represents an event (say 0: no event, 1: event A and 2: event B)

To solve it using neural networks, I frame this setting with NxM prediction, where N is the length of the signal and M the number of classes in one-hot encoding fashion. For training, I use the cross-entropy between outputs and labels; as for prediction, I just take the argmax of each time instant.

The problem is that, using three classes as in above, the network usually distinguishes well one event while it labels the rest as 0, completely neglecting the other event. The data itself seems to be well-behaved, and this pattern happens alternately.

Has anyone faced a similar situation and could spare a few tips on how to approach this?