I have over a million text documents that I would like to cluster. I used tf-idf modeling and term vector cosine for identifying similar documents in the corpus, which appeared to work well.

Some documents are annotated with issues labels (e.g., a,b,c,d – twenty in all). Each issue has as least 100 documents.

I would like to compute the average document for each cluster. What would be the best approach to normalization in this context? Right now, I am normalizing using $0.5 + \big{(}0.5 * \frac{term}{\max{term}}\big{)}$. Would it be better to compute the average of the normalized vectors instead? Alternatively, I could compute the sum of all raw term frequencies and then normalize that. What is the best approach to create a normalized term vector for each issue?

  • $\begingroup$ I got tumbleweed badge on on this one. $\endgroup$ – paparazzo Oct 22 '15 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.