Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I first use a simple function to determine the frequency of all the four and above character words. I then use this as the feature of each training sample. Now i want my algorithm to be able to cluster the training sets to according to their features, which here is the frequency of each word in the article. (Note that in this example, each article would have its own unique feature since each article has a different feature, for example an article has 10 "water and 23 "pure" and another has 8 "politics" and 14 "leverage"). Can you suggest the best possible clustering algorithm for this example?
4 Answers
I don't know if you ever read SenseCluster by Ted Pedersen : http://senseclusters.sourceforge.net/. Very good paper for sense clustering.
Also, when you analyze words, think that "computer", "computers", "computering", ... represent one concept, so only one feature. Very important for a correct analysis.
To speak about the clustering algorithm, you could use a hierarchical clustering. At each step of the algo, you merge the 2 most similar texts according to their features (using a measure of dissimilarity, euclidean distance for example). With that measure of dissimilarity, you are able to find the best number of clusters and so, the best clustering for your texts and articles.
Good luck :)
If you want to proceed on your existing path I suggest normalizing each term's frequency by its popularity in the entire corpus, so rare and hence predictive words are promoted. Then use random projections to reduce the dimensionality of these very long vectors down to size so your clustering algorithm will work better (you don't want to cluster in high dimensional spaces).
But there are other ways of topic modeling. Read this tutorial to learn more.
Cannot say it is the best one, but Latent Semantic Analysis could be one option. Basically it is based on co-occurrence, you need to weight it first.
http://en.wikipedia.org/wiki/Latent_semantic_analysis
http://lsa.colorado.edu/papers/dp1.LSAintro.pdf
The problem is that LSA does not have firm statistic support.
Have fun
One way to classify the text is by calculating the Term Frequency and Inverse Document Frequency. You can refer this paper: http://www.oracle.com/technetwork/testcontent/feature-preparation-130942.pdf
