LabelEncoder with a Multi-Layer Perceptron?

Question

So we're working on a machine learning project at work and it's the first time I'm working with an actual team on this. I got pretty good results with a model that uses the following SKLearn pipeline:

Data -> LabelEncoder -> MinMaxScaler (between 0-1) -> PCA (I go from 130 columns to 50 prime components that cover the variance) -> MLPRegressor

One of my colleagues mentioned that I shouldn't normally use LabelEncoder to encode training data, as it's meant for encoding the target variable. I did some research and now and I understand why

LabelEncoder only is not a good choice, since it brings in a natural ordering for different classes.

Then, however, my colleague mentioned that in this case it shouldn't make much of a difference as I'm using a neural net (~MLPRegressor). My question is - (if he's right - is he?) why? He basically commended me saying this usually would be a bad idea but in this case it should work.

I will try to move to one-hot encoding (currently I'm only stuck with it because I run out of memory while doing PCA on that many columns, but that's another question and I'll do some research on that separately), but for now I'd like to know if using this kind of encoding can result in inaccurately good results (I'm having an r^2 score of around 0.9 and my boss literally won't believe I achieved a result that good haha).

Answer 1

Your colleague was correct that LabelEncoder is meant for the target variable (a.k.a. labels); more-or-less equivalent OrdinalEncoder is meant for the independent variables.

IMO, your colleague is partially correct about an ordinal encoding "not making much of a difference" for a neural network. A neural net can contort its way into a correct understanding of your labels anyway, although I think an ordinal encoding will make its job substantially more difficult, so I still wouldn't advocate for it. The intervening steps, scaling and PCA, make it less clear exactly what the neural network will actually see, and I don't know if that alleviate or worsen the issue of an ordinal encoding. (Given your good score, perhaps alleviate.)