I read the book "Human-in-the-Loop Machine Learning" by Robert (Munro) Monarch about Active Learning. I don't understand the following approach to get a diverse set of items for humans to label:
- Take each item in the unlabeled data and count the average number of word matches it has with items already in the training data
- Rank the items by their average match
- Sample the item with the lowest average number of matches
- Add that item to the ‘labeled’ data and repeat 1-3 until we have sampled enough for one iteration of human review
It's not clear how to calculate the average number of word matches.