Classification when one class is other

I am working on a litigation support application using the Enron corpus, which contains about 600,000 unique text documents.

In litigation, one is often concerned with whether a document is responsive or non-responsive. One produces responsive documents to the opposing side, unless they are privileged (e.g., attorney client communication).

Here, I have sample sets of over 200 responsive and non-responsive documents.

The challenge here is that the topic of the responsive documents is about one thing, whereas the non-responsive documents could be about any number of topics, ranging from spam, to soccer practice, to business documents, etc. The non-responsive is what I would call diluted.

I don't know what those non-responsive classes are up front, and there is no purpose or value to breaking them out up front. If a document is non-responsive, then it needs to be quickly (and cheaply) dismissed.

If the samples are random when the classification is applied to the corpus my customers expect the split between corpus responsive and non-responsive to be close to the split of the sample.

What is the best approach to classify responsive versus non-responsive in this situation?

Below I briefly describe what I have tried. If there is a better approach, please share.

1. Using tf-idf, create an average vector for each document class (responsive and non-responsive).
2. Take the first 1000 terms (sorted by weight), such that each vector is the same length.
3. Normalized the vectors.
Note: these two vectors are only a 0.03 cosine similarity to each other.

4. For each document in the responsive sample set, calculate the cosine similarity to the single average vector.
Note: the average is 0.06. It is a small number, as documents in the sample set have around 40 terms, as compared to an average vector of 1000.

5. In the non-responsive perform the same analysis (compare each non-responsive sample document to the non-responsive average vector).
In the non-responsive that same comparison is 0.03. Basically, the non-responsive average vector is diluted.

Based on this approach, I cannot really conclude if document has a higher cosine similarity to responsive compared to non-responsive then it is responsive. By that measure 1/3 of the documents in the non-responsive sample set would be classified as responsive.

The non-responsive needs to be handicapped. In this approach, how do I handicap it? Is there another approach that would accomplish the same thing?

As a first step you should try logistic regression on your tf-idf vectors. It is simple to implement and would provide a good baseline for comparison. You can find an implementation in whatever language you're using. You could also try some kind of (perhaps supervised) topic modeling to create a better feature space, but that would be more involved.

• I am using .NET C# and I found an implementation but I am not following how to use it. I will study up. Oct 28, 2015 at 14:41
• I have spent some time with this and I just don't get how logistic regression on tf-idf vectors will handicap. The regression is not aware of cosine similarity. Can you provide more instructions? Oct 29, 2015 at 17:43
• Is there an inherent dependence on cosine similarity in your problem? If you are simply trying to separate responsive documents from the others, you are attempting a classification problem. One class is "responsive" and the other is "irrelevant." If you label the responsive documents as responsive == 1 and others as not == 0, then your classifier will learn to distinguish between responsive and not based on tf-idf as an input. Oct 29, 2015 at 18:01
• Yes there is an inherent dependence on cosine similarity. I am not aware how to make the logistic regression aware of that dependence. How does my classifier learn? Oct 29, 2015 at 18:22
• I'm really not sure where the requirement to use cosine similarity comes in. A logistic regression would simply separate out the two classes (which I believed to be your objective). Oct 29, 2015 at 18:37

This is more what I did then an answer to the stated question.

What I also want is a rank. Clearly a similarity of 0.9 versus 0.1 is a higher rank than 0.5 versus 0.4. My problem is 0.43 non-responsive is versus 0.45 responsive is actually non-responsive (other).

What I did for the raw score was the harmonic mean of the difference and log of the ratio. I could get .35 versus a .20 and 0.1 versus a 0.01. Using the log of the ratio seem to give absolute differences more even ground with relative differences. Then the harmonic mean kept the bigger number from dominating as much.

Then I went down the ordered by the exact ratio of the sample set and used that as the offset so a score of 0 was don't know.

This rank seems to be effective0 as the top 1/2 by rank had zero errors when looking at sample set versus the classification of the sample set. And the stuff right in the middle had the most errors.