2
$\begingroup$

I am working with a completely categorical network log data that consists of source ip address, destination ip address, source port, destination port, protocol.

Data Preprocessing performed : Converted IP addresses to integers. Removed rows with NA's. Dis not perform scaling as it doesn't make any sense with categorical data and also since I'm computing Hamming distance.

Modelling and Implementation : I have implemented 'Hamming Distance' to compute the distance matrix for the set. I have tried converting the categorical variables to numerical by converting them to factors and perform Kmeans on the data, however the accuracy I'm getting is very low as expected. I have used KModes as well and I have got cluster labels to the data, I'm having challenge proceeding further from here.

Question : Once the distance matrix is computed, how can I go about detecting the outliers with such data type? I really appreciate any recommendations. Thanks in advance

Data Snippet :  2887562076 2344654028        6   41940      80
                2344654028 2887562076        6      80   41940
                2344654028 2887562076        6      80   41940
$\endgroup$
1
  • 1
    $\begingroup$ Converting IP addresses to integers makes no sense. 254.x.x.x is no closer to 255.x.x.x than to 122.x.x.x. You should treat them as categorical values, but they have a way too high cardinality. You need to featurize this in a different way. $\endgroup$ Jul 21, 2017 at 12:34

1 Answer 1

1
$\begingroup$

Foremost, in my opinion categorical data should not be converted to continuous format at all. This is statistically incorrect.

Second, if a categorical data say Sex with two levels Male, Female exists then it can be written as 1,2, but, it should be specified that these numbers are to be treated as nominal and not continuous.

Continuing further, one method of detecting outliers in categorical data is to calculate the frequency of the data points and then average each such obtained frequency to the total number of data points. The data points with the minimum frequencies can be considered as outliers.

You might find this paper and this SO post implementation in R interesting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.