0
$\begingroup$

For example lets say I have a dataset with independent features age, gender, name, and income. While my dependent variable is load approval status.

If I want to use KNN or SVM, do I need to convert gender into a numerical variable? and if yes, how is it used in algorithms such as KNN or SVM which are distance based algorithms?

$\endgroup$
1
$\begingroup$

The traditional way of dealing with categorical variables is to one-hot encode these. One hot encoding will transform one categorical variable with n values into n different features which values can be only 1 or 0, 1 indicating the presence of the corresponding category and 0 the absence.

As an example, instead of having:

---------------
|id  |Category|
---------------
| 0  |    A   |
---------------
| 1  |    B   |
---------------
| 2  |    A   |
---------------

You would have:

--------------------------------
|id  | Category_A | Category_B |
--------------------------------
| 0  |     1      |     0      |
--------------------------------
| 1  |     0      |     1      |
--------------------------------
| 2  |     1      |     0      |
--------------------------------

Distance based algorithms then use the vectors representing the features to compute distances, whether it is the euclidean, cosine, hamming distance... After one hot encoding, you can see that these vectors are exclusively numeric, the 'categorical' aspect being translated into the addition of new dimensions.

With one hot encoding, there are a few common things you must watch out for:

  • It is recommended to drop the last column created by one hot encoding, because it can be inferred from the others. As an example, if you have a feature F taking three possible values A, B and C, one hot encoding would produce three columns F_A, F_B and F_C. You have F_C = not(F_A + F_B), i.e. 'if the value is neither A nor B then it is C'. In this case you should keep only columns F_A and F_B (or F_A and F_C , F_B and F_C whatever) since the third column is redundant and adds therefore no information. Keep in mind that if you F allows the presence of missing values, this has to be taken in account by either imputing the values, or considering the 'missing value' as a fourth possible category.

  • One hot encoding augments the dimension and the sparsity of the dataset. This can also harm the performance of the algorithm, especially when you have a lot of categorical variables with a lot of possible values.In order to avoid that, you can apply dimensionality reduction (as an example with PCA) to the one hot encoded dataset, and/or perform feature selection.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.