How to deal with categorical variables

Question

I am new to Data Science and have a problem with categorical variables. My data set has 2 columns of strings - Departure City and Arrival City. It looks like this:

ID  | DepCity| ArrCity |Price|Time|SomeColumn   |ColToPredict|
1   |London  | Berlin  | 300 | 95 | 220         | 4          |
2   |Dublin  | Nice    | 420 |115 | 59          | 1          |
3   |Milan   | Brussels| 150 |108 | 154         | 3          |
4   |Paris   | Rome    | 160 |120 | 200         | 4          |
250 |Madrid  | Oslo    | 290 |300 | 110         | 2          |

So there are a lot of categorical variables in both columns and these columns are important (their values depend on other columns).

I use Python and sklearn. And it's not possible just to eliminate them as suggested in some tutorials.

I know there is a way to deal with categorical variables by creating new columns with zeros and ones for each variable. But I'm not sure that it's my case, because I have about 30 unique cities in each column. What will be the best way to convert categorical variables into numerical?

Answer 1

In your specific case you are working with cities, which can be modelled as geographic location. I've never used it be you could look at the geohash method.

Or could also use an identifiant for each city. An algorithm like a random forest can directly use it this way.

Answer 2

Creating new columns for each variable with 0s and 1s is a technique called "One Hot Encoding" and I think it's fair to say that it is a standard way to treat categorical variables when making use of algorithms that cannot use categorical variables. As you correctly identify, it can cause problems by drastically increasing the dimensionality of the problem, especially if you have many levels in your categorical variables. Other encoding techniques commonly employed, such as helmert, equally increase the dimensionality.

An alternative encoding is binary coding. In binary coding, the category levels are first encoded as ordinal (numbered sequentially). The ordinal codes are converted into binary and the binary digits are split into separate columns. To encode 30 unique cities, you would need 5 columns. The dimensionality is increased by the number of digits required to represent the number of levels in binary, for each categorical variable.