Say you've got two inputs (X1 and X2) that you want to use to predict Y.

You're not sure how important X1 and X2 are for predicting Y, but you assume about even.

One-hot encoding is a good strategy for X1 and it yields a vector of size 10,000.

X2 is an unsigned int between 0 and 1, so you can just pass it as-is.

So your network will look something like

10,001 -> {some hls} -> Out-layer

In theory it could "learn" to assign a lot of importance to X2 compared to X1, but in practice I assume this is hard when the difference in dimension varies so much. At least based on some datasets I tested this was certainly true.

The one "simple" solution I can think of to this issue is instead having a network shapes more like:

10,000 -> {some hls} -> 1,000 --
                               |-> {some hls} -> Out-layer
1 -> {some hls} ------> 1,000 --

So basically have some encoders/backbones/whatever-you-want-to-call-them, that increase/reduce the size of certain groups of inputs and train them at the same time as the normal network.

My question here is:

a) Is the problem I identified here a "real" one or would it not come up in practice ? b) Does it have a name and are there already established solutions to it ? c) Is the solution I proposed here a "good one" ? d) Do you have any examples of networks where this method is actually being used ? Preferably implemented in pytroch but it doesn't matter really, the implementation per-say seems easy enough. f) Are there potential pitfalls in terms of performance and or implementation to this solution that might not be obvious ?


The main problem I see here is that OHE is almost never a good idea with that many categories. With neural networks you will usually get better performance by using embeddings. So instead of

X1 -{OHE}-> 10,000 -> {..} -> 1,000 you could go straight to X1 -{embedding}-> 50, where the embedding dimension should probably a lot lower than 1,000.

Either way, more important is that one calculates feature importance in a coherent framework. The easiest one is probably permutation importance (another could be shap values), which basically comes down to (repeatedly) shuffling the input features X1 and X2 and record the impact on Y. This works also for OHE variables if you make sure to shuffle X1 before one-hot encoding (and not shuffling the 10,000 encoded vectors independently). In such a framework X2 doesn't need to be "magnified" for a fair comparison with X1.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.