Anomaly detection in nominal big data

Question

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is there any way to transform this nominal values into numeric ones in a way that the algorithm will work?

I've used preprocessing.LabeledEncoder(), but then when I apply the algorithm it finds anomalies - the values that most differ from the mean it seems.

Are there any examples of an algorithm or another way to transform the data?

Answer 1

You would have to post more info about your data and maybe even some samples, but have you considered using tokenization for the data? It's from the NLP world and would allow you to assign a numeric value for each bit of you "nominal" data.

Answer 2

Preprocessing.LabeledEncoder() is to encode the target y only accordinf to scikit doc. To encode Nominal Feature you should use sklearn.preprocessing.OneHotEncoder. But beware if unique values are too numerous, this may be a problem.

doc here: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder

example here: https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py

Answer 3

My understanding is you have a categorical variable that could take on one out of 10,000 possible values.

That is probably too high of cardinality for scikit-learn's OneHotEncoder to be useful. There are many other possible category encoders.

Given the high cardinality, an useful option might be using a pre-trained word embedding space such as OpenAI embeddings. The embedding space would provide numeric values that are very amenable to anomaly detection.