Say we have 100,000 documents tagged with 10,000 different tags (Max 5 tag per document). We wish to limit allowed tags to a list of 500 tags.
How to select 500 tags in order to cover the largest set of documents ?
Firstly, I chose the 500 most frequent tags. If we keep only these 500 tags, 70% of documents keep at least 1 tag.
I'm looking for a better method to pick the 500 most pertinents tags.
my best set of 5 tags to cover all documents is [italian, german, french, chineese, japan].
it is not [italian, spanish, indian, finnish, english] although these are the most frequent tags.
I tried a Single Value Decomposition on the matrix : [documentID, set of tags]. but after that ? is it a good idea ? how to get 500 tags from the SVD results ?