1
$\begingroup$

I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies method in python and the NominalToBinary method in WEKA, but the problem is that some nominal features contain 64 values so the conversion increases the dimensionality of the data a lot, and this can create problems for the classifier.

My question is if I can convert the nominal attributes by establishing a correspondence between each category of a nominal feature and a sequence of integer values, for example protocol_type {tcp=0, udp=1, icmp=2...etc}? Would this alter the credibility of the resulted data set?

$\endgroup$
3
$\begingroup$

By converting a nominal attribute to a single numeric attribute as you described, you are implicitly introducing an ordering over the nominal labels which is a bad representation of the data, and can lead to unwanted effects from a classifier. Does it make sense to say that UDP should be inbetween TCP and ICMP? (no!) Imagine you are training a $k$-NN model on this data. It doesn't make sense to say that ICMP should be "further away" from TCP than UDP, but if you adopted the mapping that you suggested, the representation of the data has this assumption built-in. Alternatively, what if you are training a decision tree-based model? Usually, in decision trees, binary split points are chosen for numeric attributes. There could be some randomness in your training data where splits at certain values of the numeric attribute results in overfitting to noise.

Typically when converting a nominal attribute to numeric, one numeric attribute per nominal label is created. Each attribute is set to one if the corresponding nominal label is set, and zero otherwise. For example, if a nominal attribute called protocol has labels {tcp, udp, icmp}, then this dataset:

$$ \begin{array}{ccl} \text{inst.} & \text{protocol} & \text{other attributes} \\ \hline 1 & \text{tcp} & \dots \\ 2 & \text{icmp}& \dots \\ 3 & \text{icmp}& \dots \\ \vdots & \vdots & \ddots \end{array} $$

could be converted as follows:

$$ \begin{array}{ccccl} \text{inst.} & \text{tcp} & \text{udp} & \text{icmp} & \text{other attributes} \\ \hline 1 & 1 & 0 & 0 & \dots \\ 2 & 0 & 0 & 1 & \dots \\ 3 & 0 & 0 & 1 & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \\ \end{array} $$

This is what the NominalToBinary filter does in WEKA. As you mention, the downside of this is that a large number of additional attributes can be introduced if the number of distinct nominal values is high.

If the dimensionality is too high after the conversion, you may want to consider using a dimensionality reduction technique such as random projection, PCA, t-SNE, etc. Note that this will reduce the interpretability of your model. You could also use feature selection techniques to remove some of the less useful attributes. It is possible that some of the nominal labels are not useful for your model, and you will improve performance by removing them. Another thing you could try is to use your domain knowledge to reduce the number of categories. For example, TCP and UDP are both transport protocols, maybe for your application the distinction between TCP and UDP is not that important and you can put instances with protocol $\in$ {tcp, udp} into a new category, removing the old ones.

$\endgroup$
3
  • $\begingroup$ thank you so much for your help, even if i have just a litle objection concerning your sentence "Does it make sense to say that UDP should be inbetween TCP and ICMP? (no!)", cause i meant to represent each nominal value with a given numeric value, i mean where the problem can occurs? thank you again $\endgroup$ – user4309930 Oct 28 '17 at 19:47
  • $\begingroup$ By representing each nominal value with a given numeric value, you are imposing an ordering on the nominal values. If your model doesn't treat the new numeric attribute as a continuous variable, then it's the same as if it were a nominal attribute. I've added some more clarification as to exactly how this can create poor performance in the model in my answer. $\endgroup$ – timleathart Oct 29 '17 at 22:07
  • $\begingroup$ Now i get it, thank you so much for your help and your time ;) $\endgroup$ – user4309930 Oct 29 '17 at 22:36
2
$\begingroup$

For encoding of the categorical variables with high cardinality (i.e. with large number of levels) you may want to try the so called impact coding.

The main idea is very simple, you just split the dataset into non-overlapping buckets by the variable of interest (“protocol” in your case) and calculate average of your response variable over each bucket. Then, the values of the categorical variable can be substituted by the average value over particular bucket.

Avg(response | protocol=”tcp”)

Avg(response | protocol=”icmp”)

Avg(response | protocol=”udp”)

The tricky part is to avoid data leakage, this can be done by splitting the entire dataset into several subsets (e.g. “encoding”, “training”, “validation”,…) and using only the data from “encoding” dataset for the nominal-to-numerical conversion.

I’ve learned about this approach from Win-Vector blog and their paper: vtreat: a data.frame Processor for Predictive Modeling, which I highly recommend.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.