0
$\begingroup$

Suppose I had a simple linear regression model that had the following input or X variable:

[North]
[East]
[West]
[South]
[North, East]
...
[North, East, West, South]

and I decided to numerate them like:

[North] - 0
[East] - 1
[West] - 2
[South] - 3
[North, East] - 4
...
[North, East, West, South] - 15

I had someone take a look at my model and tell me to use One Hot Encoder or One Hot Binary Encoder instead of assigning inputs like this.

My question is from a linear regression perspective what is the advantage for using OHE over my simple numerical mapping? If we can quantify an accuracy loss would it be substantial? If I had 10 model variables that I had to map like this would the loss be more substantial?

I want to know what sacrifice I'm making not using OHE

$\endgroup$
1
$\begingroup$

The issue with numerical encoding in this context is you are enforcing that your input variable X is ordinal when it's likely not. This is telling your model that the order in which you encode your inputs are either increasing or decreasing monotonically with your target. Let's say you encoded your data like this:

[North] - 0
[East] - 1
[West] - 2
[South] - 3
[North, East] - 4
...
[North, East, West, South] - 15

If you train a linear regression model with this encoding you are telling your model that [North] either indicates a higher or lower target than [North, East, West, South], and that [East], [West], [South], and [North, East] are somewhere in between. What if [West] typically has a lower target than either [North] or [North, East, West, South]? In this case you would be enforcing some constraint on your model which is not true.

To test this you could split your input data in a 70/30 train-test-split and evaluate how your numerical encoding performs against one-hot-encoding on the test set. Another alternative would be to look into Target Encoding - this would allow you to keep a small feature space (an issue with one-hot-encoding) while attempting to keep your input and target increasing/decreasing monotonically. I would recommend testing all three encoding methods to figure out the ideal solution for your given problem.

$\endgroup$
2
  • $\begingroup$ Thank you very much for your answer. I just have one question about your answer. If the one hot encoding generated incremental values (e.g. [North] is 01, [East] is 10, [North East] is 100 would the same outcome happen? Since the values scale up the same? $\endgroup$ – Sad CRUD Developer Dec 9 '20 at 3:52
  • 1
    $\begingroup$ The issue isn't with the scale of the encoding - its the order in which you are encoding. encoding {1, 2, 3, 4} would have the same theoretical issue as encoding {1, 10, 100, 1000}. $\endgroup$ – Oliver Foster Dec 10 '20 at 0:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.