I have over a million text documents that I would like to cluster. I used tf-idf modeling and term vector cosine for identifying similar documents in the corpus, which appeared to work well.

Some documents are annotated with issues labels (e.g., a,b,c,d – twenty in all). Each issue has as least 100 documents.

I would like to compute the average document for each cluster. What would be the best approach to normalization in this context? Right now, I am normalizing using $0.5 + \big{(}0.5 * \frac{term}{\max{term}}\big{)}$. Would it be better to compute the average of the normalized vectors instead? Alternatively, I could compute the sum of all raw term frequencies and then normalize that. What is the best approach to create a normalized term vector for each issue?

  • $\begingroup$ I got tumbleweed badge on on this one. $\endgroup$
    – paparazzo
    Commented Oct 22, 2015 at 20:32


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.