1
$\begingroup$

When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have a feature vector with values A, B and c. The first method will transom A,B and C to numeric values such 1,2 and 3 respectively, other researches use (1,0,0), (0,1,0) and (0,0,1).

What is the difference between the first method and the second one?

The only difference I can think of is, if you use binary values, the size of the training/testing data will increase linearly according to how many values you have, which may slow down the performance, while the first one will keep the size unchanged.

Does either of these methods will effect the accuracy of your machine learning model (or classifier)?

$\endgroup$
2
$\begingroup$

While using one-hot (binary) encoding certainly takes more space, it also implies an independence assumption among the data. On the other hand, using integers such as 1, 2 and 3 implies some kind of a relationship between them.

The problem that you mention of linear increase in size with one-hot encoding is common and can be treated by using something such as an embedding. An embedding also helps define a sense of distance among different datapoints.

https://en.wikipedia.org/wiki/Word_embedding

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your answer, so which one you will recommend? (I mean if there is no relationship between each value). $\endgroup$ – U. User Nov 4 '18 at 17:44
  • $\begingroup$ One hot or an embedding. $\endgroup$ – Anshul G. Nov 4 '18 at 20:58
2
$\begingroup$

The numbers shows a relationship i.e. when you use numerical values inplace of text data it means one value is higher than the other. Let's say you are taking nominal values i.e. (Red, Blue, Green) and represent it using (1 , 2 , 3) . Your model will consider it as 3>2>1 but in general we are using colours which do not say that Red>Blue>Green. So instead we prefer One Hot encoding which creates dummy variable and uses 1/0 value to represent them.

Although if your prefer ordinal variables i.e. High, Medium, Low .Then these values can be represented using number because it does show an order which is 3>2>1. It can be encoded using label encoder or by mapping in an order.

| improve this answer | |
$\endgroup$
0
$\begingroup$

if your categorical variable has an order so use numerical and if there isn't any order use binary. when you use numerical type it has some meaning so be careful.

some algorithms can handle lots of variables together. also maybe you can merge some hot encode variable if they are very rare or for showing 3 value you can use just two binary variables.

if you need more explanation let me know.

| improve this answer | |
$\endgroup$
  • $\begingroup$ So if I have categorical variables that don't have any order and I used numerical type encoding, will this influence the accuracy and precision of my model ? $\endgroup$ – U. User Nov 4 '18 at 19:54
  • $\begingroup$ if it is without order use binary encoding. $\endgroup$ – parvij Nov 5 '18 at 5:01
  • 2
    $\begingroup$ Even if there is some "order" to categorical values, it is often still necessary to use one hot encoding as there is an implied numerical or spatial relationship between two integers that may not be implied in the ordinal data. E.g. 2 is 100% larger than 1, but 3 is only 50% larger than 2. This can lead to issues in many models. $\endgroup$ – Skiddles Nov 6 '18 at 19:59
  • $\begingroup$ I agree with @Skiddles, some algorithm is sensitive to this issue $\endgroup$ – parvij Nov 7 '18 at 19:50
  • $\begingroup$ Can you give an example of such algorithms ? $\endgroup$ – U. User Dec 4 '19 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.