# Identify outliers for annotation in text data

I read the book "Human-in-the-Loop Machine Learning" by Robert (Munro) Monarch about Active Learning. I don't understand the following approach to get a diverse set of items for humans to label:

1. Take each item in the unlabeled data and count the average number of word matches it has with items already in the training data
2. Rank the items by their average match
3. Sample the item with the lowest average number of matches
4. Add that item to the ‘labeled’ data and repeat 1-3 until we have sampled enough for one iteration of human review

It's not clear how to calculate the average number of word matches.

• For every document $$d_u$$ in the unlabeled data, count the number of words in common with every document $$d_l$$ in the labeled data. This value is the "match score" between $$d_u$$ and $$d_l$$.
• As output from the above step, for a single document $$d_u$$ one obtains a "match score" for every labeled document. The average across the labeled documents gives the "average match" for $$d_u$$.