I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic.
I have used a simple method of mining documents related to a specific topic based on a keyword list. Basically, if the document contains one of the words from the keyword list it will retrieve it. If that particular word could be used in a different context it checks the post again for other associated words which would usually be found in similar types of documents. This is a simple method but seems to work well, and can be applied to any topic quickly and easily. The main detractor seems to be the keyword lists need to be created and maintained which can be time consuming and inefficient.
In recent times machine learning methods have been used for this type of document classification. It seems this method is able to judge "context" better in documents but requires large datasets to be trained on and also require continual training if new data needs to be classified.
It feels like people dismiss lexical methods since the emergence of machine learning methods but is this warranted? It seems like lexical methods can still get good results, especially on small documents which don't contain much context.
What are the pros and cons of each?