Can I use unsupervised learning followed by supervised learning?

Question

I have a question about classifying documents using supervised learning and unsupervised learning.

For example: - I have a bunch of documents talking about football. As we know, football has a different meaning in UK, USA, and Australia. Therefore, it is difficult to classify these documents to three different categorizations (which are soccer, American football and Australian football).

My approach tries to use cosine similarity terms based on unsupervised learning. After we use the cluster learning, we are able to create a number of clusters based on cosine similarity, where each cluster will contain similar documents terms. After we create the clusters, we can use a semantic feature to identify these clusters depending on a supervised model like SVM to make accurate categorizations.

My goal is to create more accurate categorizations, because if I would like to test a new document, I would like know if this document can be related to these categorizations or not.

Answer 1

You can definitely try to first cluster your data, and then try to see if the cluster information helps your classification task.

For example if your data looked like this (in 1D):

AA A AA A A      BBB B B B BB BB BB      AA AA A A AAA

then it may be reasonable to run a clustering algorithm on each class, to obtain two different kinds of A, and learn two separate classifiers for A1 and A2, and just drop the cluster distinction for the final output.

Other common unsupervised techniques used include PCA.

As for your football example, the problem is that the unsupervised algorithm does not know what it should be looking for. Instead of learning to separate american football and soccer, it may just as well decide to cluster on international vs. national games. Or Europe vs. U.S.; which may look like it learned about american football and soccer at first, but it put american soccer into the same cluster as american football, and american football teams in Europe into the Europe cluster... because it does not have guidance on what structure you are interested in; and the continents are a valid structure, too!

So usually, I would not blindly assume that unsupervised techniques yield a distrinction that matches your desired result. They can yield any kind of structure, and you will want to carefully inspect what they found before using it. If you use it blindly, make sure you spend enough time on evaluation (e.g. if the clustering improves your classifier performance, then it probably worked as intended ...)

Answer 2

It sounds as if you want to use unsupervized learning to create a training set. Am I right? You use your cluster analysis to determine which docs come from UK, US or Oz -- or which docs are talking about Soccer, Football or Australian football respectively? Then feed those tagged docs into a supervized learning algorithm of some sort?

How well this works will depend entirely on how well you can distinguish UK, US and OZ. I would have thought it would be fairly straightforward to find documents where national origin was known, so that you could build a supervized algorithm for detecting language variant. You wouldn't even need a corpus that talked about football, since dialectical differences show up in other ways that are subject matter independent. (For example, I am clearly from North America, since I just wrote "in ways that are subject matter independent" rather than "Since dialectical differences do not depend on subject matter").

However, the answer to your question, "can I use unsupervized learning and then supervized learning" is No, if you are looking for supervized learning. If the results of an unsupervized learning algorithm are fed to a supervized learning algorithm, the net result is unsupervized --- there are still no grown-ups in the room. And the classification errors of the resulting process will contain error terms from both stages. You won't get the same performance as you would if you did a SVM with properly tagged training data. This doesn't mean you shouldn't use the method you propose ... it might still work well ... but it won't be a supervized learning algorithm.

Answer 3

Using unsupervised learning to reduce the dimensionality and then using supervised learning to obtain an accurate predictive model is commonly used. See for example Bhat and Zaelit, 2012 where they first use PCA to reduce the dimension of a problem from 87 to 35. Then, they use L1 regression to obtain the best predictive model. This method beats non-linear tree based models built on the entire dataset and also its subset.

If your goal is to create more accurate classification of data into clusters, then a commonly used technique is to use supervised learning as a method to accurately pick the number of clusters see Pan et al, 2013 for a recent example. The basic approach here is to choose the number of clusters such that a supervised multi-class method can learn these clusters and predict the clusters with the highest out of sample accuracy. This is one way to convince yourself that the clusters are both meaningful and predictable.

Another approach, if your goal is to classify documents as being from US/ USA/ Australia or for that matter discussing, soccer/ American football/ Australian football could be to solve three binary classification problems that independently predict if the document talks about soccer, American football or Australian football. Combining the results from these three classifiers (known as binary relevance), you could also have the ability of tagging a document as both soccer or American football or any combination of the above three tags.

Answer 4

I don't perfectly understand the supervised part of the question but it might be useful to note that:

• the unsupervised classification of documents seems to correspond to the NLP task of topic modeling, a task for which there are good methods and tools available.
• in the proposed example about football, the polysemy of the target word calls for word sense discrimination.

The two tasks and the methods to solve them are quite similar.