I have a classification model (DNN/Linear layers with some transformers and other things later). The input to the model are several different modalities of different lengths and different amounts of information. I am trying to mitigate the dimensionality difference by projecting different modalities into the same dimensional space and then combining them into the same space. However, the model seems to be focusing completely on the strongest input features and almost completely ignores the abundance of weaker ones. All of the input features are normalized (bools to 0/1 and continuous to mean=0, stddev=1), so the issue is not the scale of feature values but the predictive power of those few features which end up choking others.
Are there any methods out there for addressing this?