1
$\begingroup$

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is there any way to transform this nominal values into numeric ones in a way that the algorithm will work?

I've used preprocessing.LabeledEncoder(), but then when I apply the algorithm it finds anomalies - the values that most differ from the mean it seems.

Are there any examples of an algorithm or another way to transform the data?

$\endgroup$

3 Answers 3

0
$\begingroup$

You would have to post more info about your data and maybe even some samples, but have you considered using tokenization for the data? It's from the NLP world and would allow you to assign a numeric value for each bit of you "nominal" data.

$\endgroup$
0
$\begingroup$

Preprocessing.LabeledEncoder() is to encode the target y only accordinf to scikit doc. To encode Nominal Feature you should use sklearn.preprocessing.OneHotEncoder. But beware if unique values are too numerous, this may be a problem.

doc here: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder

example here: https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py

$\endgroup$
0
$\begingroup$

My understanding is you have a categorical variable that could take on one out of 10,000 possible values.

That is probably too high of cardinality for scikit-learn's OneHotEncoder to be useful. There are many other possible category encoders.

Given the high cardinality, an useful option might be using a pre-trained word embedding space such as OpenAI embeddings. The embedding space would provide numeric values that are very amenable to anomaly detection.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.