I have a question about classifying documents using supervised learning and unsupervised learning.

For example: - I have a bunch of documents talking about football.
As we know football has different meaning in UK, USA and Australia. Therefore, it is difficult to classify these documents to three different categorizations which are soccer, American football and Australian football.
My approach tries to use cosine similarity terms which is based on unsupervised. After we use the cluster learning, we are able to create a number of clusters based on cosine similarity which each cluster will contain similar documents terms. After we create the clusters, we can use a semantic feature to identify these clusters depend on supervised model like SVM to make accurate categorizations.

My goal is to create more accurate categorizations because if I want to test a new document I want know if this document can be related to these categorizations or not.

  • 2
    $\begingroup$ Sure, you can. It's common to do dimensionality reduction as a preprocessing step. $\endgroup$ – Emre Aug 16 '14 at 17:39
  • 2
    $\begingroup$ I don't get the point of the question. What would prevent you from doing such a thing? The data science police? (I don't mean to be disrespectful, I simply don't understand the reason for the question). $\endgroup$ – Trylks Aug 28 '14 at 18:20
  • $\begingroup$ The point of the question is clearly "will the results be meaningless," and the answer is, "They very well might be." If the clusters exist because of uninformative features, then the supervised method will only be learning those same uninformative features. $\endgroup$ – one_observation May 14 '18 at 16:07

You can definitely try to first cluster your data, and then try to see if the cluster information helps your classification task.

For example if your data looked like this (in 1D):


then it may be reasonable to run a clustering algorithm on each class, to obtain two different kinds of A, and learn two separate classifiers for A1 and A2, and just drop the cluster distinction for the final output.

Other common unsupervised techniques used include PCA.

As for your football example, the problem is that the unsupervised algorithm does not know what it should be looking for. Instead of learning to separate american football and soccer, it may just as well decide to cluster on international vs. national games. Or Europe vs. U.S.; which may look like it learned about american football and soccer at first, but it put american soccer into the same cluster as american football, and american football teams in Europe into the Europe cluster... because it does not have guidance on what structure you are interested in; and the continents are a valid structure, too!

So usually, I would not blindly assume that unsupervised techniques yield a distrinction that matches your desired result. They can yield any kind of structure, and you will want to carefully inspect what they found before using it. If you use it blindly, make sure you spend enough time on evaluation (e.g. if the clustering improves your classifier performance, then it probably worked as intended ...)


It sounds as if you want to use unsupervized learning to create a training set. Am I right? You use your cluster analysis to determine which docs come from UK, US or Oz -- or which docs are talking about Soccer, Football or Australian football respectively? Then feed those tagged docs into a supervized learning algorithm of some sort?

How well this works will depend entirely on how well you can distinguish UK, US and OZ. I would have thought it would be fairly straightforward to find documents where national origin was known, so that you could build a supervized algorithm for detecting language variant. You wouldn't even need a corpus that talked about football, since dialectical differences show up in other ways that are subject matter independent. (For example, I am clearly from North America, since I just wrote "in ways that are subject matter independent" rather than "Since dialectical differences do not depend on subject matter").

However, the answer to your question, "can I use unsupervized learning and then supervized learning" is No, if you are looking for supervized learning. If the results of an unsupervized learning algorithm are fed to a supervized learning algorithm, the net result is unsupervized --- there are still no grown-ups in the room. And the classification errors of the resulting process will contain error terms from both stages. You won't get the same performance as you would if you did a SVM with properly tagged training data. This doesn't mean you shouldn't use the method you propose ... it might still work well ... but it won't be a supervized learning algorithm.


Using unsupervised learning to reduce the dimensionality and then using supervised learning to obtain an accurate predictive model is commonly used. See for example Bhat and Zaelit, 2012 where they first use PCA to reduce the dimension of a problem from 87 to 35. Then, they use L1 regression to obtain the best predictive model. This method beats non-linear tree based models built on the entire dataset and also its subset.

If your goal is to create more accurate classification of data into clusters, then a commonly used technique is to use supervised learning as a method to accurately pick the number of clusters see Pan et al, 2013 for a recent example. The basic approach here is to choose the number of clusters such that a supervised multi-class method can learn these clusters and predict the clusters with the highest out of sample accuracy. This is one way to convince yourself that the clusters are both meaningful and predictable.

Another approach, if your goal is to classify documents as being from US/ USA/ Australia or for that matter discussing, soccer/ American football/ Australian football could be to solve three binary classification problems that independently predict if the document talks about soccer, American football or Australian football. Combining the results from these three classifiers (known as binary relevance), you could also have the ability of tagging a document as both soccer or American football or any combination of the above three tags.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.