1
$\begingroup$

I'm working on a problem of anomaly detection, where at the end of the anomaly detection I will have a group of documents consisting of a title of each object that was flagged as anomalous.

At the same time I have another group of documents which are the documents/texts of a title of each object that was not flagged as anomalous.

anomalous_titles =[[Product:A - sub_group:X1 - pod: P1 - function: M1], [Product:B...],..]
not_anomalous_titles =[[Product:R - type:TX - producer: XX], [Product:B...],..]

What I would like to do here is to understand if there are any words or patterns that is shared among the anomalous documents which are not common in the group of non anomalous documents.

What method would be good to apply in this scenario? I know about TF-IDF and Topic Modelling, but I don't know if it makes sense for this use case?

Appreciate any input!

$\endgroup$

1 Answer 1

1
$\begingroup$

TF-IDF and Topic Modelling wouldn't be suitable as they do not take classes into account. One approach would be to train a basic classifier and extract important features per class.

The steps:

  1. Create a TF-IDF matrix for the text corpus.
  2. Train a basic classifier using the TF-IDF Matrix as feature matrix and the classes as target. (A decent accuracy is enough.)
  3. Get the feature_importances from the trained classifier.
  4. Sort to get most important features and their corresponding classes.
import numpy as np
from collections import defaultdict
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier

# Loading sample data
categories = ['comp.sys.mac.hardware', 'rec.autos', 'sci.space', 'rec.sport.baseball']
newsgroups = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'),categories=categories)

# 1. Fit corpus to tfidf vectorizer
tfidf = TfidfVectorizer(min_df=15, max_df=0.95, max_features=5_000)
tfidf_matrix = tfidf.fit_transform(newsgroups.data)

# 2. Train classifier
clf = RandomForestClassifier()
clf.fit(tfidf_matrix, newsgroups.target)

# 3. Get feature importances
feature_importances = clf.feature_importances_

# 4. Sort and get important features
word_indices = np.argsort(feature_importances)[::-1] # using argsort we get indices of important features
feature_names = tfidf.get_feature_names() # Lookup to get words from index

top_n = 50 # Top N features to be considered
top_words_per_class = defaultdict(list)
for word_idx in word_indices[:top_n]:
    word = feature_names[word_idx]
    word_class = newsgroups.target_names[clf.predict(tfidf.transform([word]))[0]]
    top_words_per_class[word_class].append(word)

The top_words_per_class would be like:

{
  "rec.autos": ["car", "cars", "engine", "ford", "like", "dealer", "oil", "toyota"],
  "sci.space": ["space", "nasa", "orbit", "launch", "earth", "moon", "shuttle", "thanks", "program", "project", "spacecraft"], 
  "comp.sys.mac.hardware": ["mac", "apple", "drive", "scsi", "centris", "video", "quadra", "monitor", "se", "card", "powerbook", "use", "problem", "simms", "software", "modem"],
  "rec.sport.baseball": ["baseball", "game", "team", "games", "season", "players", "year", "league", "runs", "hit", "player", "braves", "teams", "pitching"]}
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.