I'm looking for a method of allocating documents (30K and growing) to a set of some 200 categories.
The categories will be user defined and will grow over time.
As my data is unlabelled my thought process is to try an build a system that aids in the manual classification by providing a rough first pass classifier. User can then quickly go through the classified documents and accept/reject the classifications.
Once we have a sufficient large set of classified documents then I'm hoping to use some AI system to automatically classify the remaining documents and new ones as they are published.
I've had a play with LDA but as I understand it, LDA essentially chooses the topics which don't necessarily map to the categories that I want to define.
I've have also built a rule engine that allows me to manually define common key words to map each document to a category but this is still a fairly manual process as I need to define keywords for each of the categories.