Do I need a multilabel classification machine learning methodology or is it unnecessary?

Question

Introduction
I’m working on a social science research project that involves a Natural Language Processing methodology. I’m assigning multiple labels (For example, label 1: Blockchain, Label 2: Democracy) to news articles that describe their content. I’m then running them through a couple of Python sentiment packages and I’ll be analysing changes in the correlation between different labels over time and what their sentiment looks like.

For example, if I retrieve articles from 2006-2016, i’d expect to see nothing about Blockchain until 2008, then I’d expect to see a growing number of blockchain related articles. After the launch of the Ethereum Blockchain in 2014, I’d expect to see a growing correlation between blockchain labels and democratisation labels because the Ethereum virtual machine facilitates “decentralized finance”, accessibility to markets, technology etc etc.

One of my learning objectives is to bolster my Python knowledge but I need to justify any research design decisions. I don’t want an unnecessarily complicated methodology for the sake of using elaborate Python coding and fancy sounding libraries.

Question
For this project, is the use of supervised machine learning meaningful/justified/necessary or should I keep things simple? Please can you provide thoughts on the below considerations? Anything else I should bring into this assessment?

Background
Simple Approach to Multilabel Classification

• Two labels: 1. Blockchain 2. Democracy.
• Create lists of terms associated with each of these labels. For example, the “Democracy” label might include: Democra (which would pull democracy, democrat, democratisation etc.), vote, voting, liberal, presidency, parliament, representative government, etc.
• Filter my entire list of 5000 news articles for text that contains these terms, tag the articles if they contain any single instance of any of the terms. Tag each article with multiple labels if they contain terms relating to either label.
• Manually audit the results by randomly selecting 100 articles, reading them in full and judging the labels. This may result in further manual correction, for example, if the term “liberal” returns numerous articles when the term is not used in the context of democracy, I might then decide to manually check the context of all instances for that term..
• Pros and Cons: it will be impossible to manually review my filters in full. The existence of one instance of a term (an article mentions the word “democracy” once) doesn’t necessarily mean an article is about that subject (the threshold for receiving a particular label is very low). All terms will be considered unlike with the machine learning process where the training set might not include every term. The filtering/labelling of articles could be automated and is therefore scalable.

Advanced Approach to Multilabel Classification

• Two labels: 1. Blockchain 2. Democracy
• Create a dataset for training and testing a multilabel classification prediction algorithm: Manually tag a subset of 500 news articles (10% of the total population) with the two labels. This would be done using the simple approach described above combined with a manual audit of labels that this produces. For example, if the training/testing subset consists of 500 articles, I’ll read 100 of them to explore nuances and correct the labels. This may result in the decision to manually audit all 500.
• Run a TFIDF function to assess the importance of all words in each article
• Use binary relevance to assess each label independently with a Naive Bayes Algorithm for the classification.
• If the testing yields decent accuracy results, then use the model for the remaining 4500 articles
• Pros and Cons: this process will assess the importance of all words in each article and more broadly consider phrases that include my search terms. The labelling of this test/training data will be of higher quality than the labelling of all 5000 articles in the simple approach (manual review time focussed on 500 articles rather than 5000). If the training data doesn’t include particular terms however, then they will be dropped from the prediction process. Once the model is adequately trained, I can scale the size of this research without incurring additional work. For example, I could run 10,000 articles through the prediction algorithms.

My preliminary answer and chosen approach
Multilabel classification prediction algorithm (machine learning) not necessary but advantageous for the following reasons:

• the importance of all words will be assessed
• The labelling of training data can be thoroughly reviewed manually
• The main downside is that the training data might not include all terms considered relevant to a particular topic but I can address this by inserting articles about those topics or by fabricating them.
• I appreciate there are levels to the multilabel classification problem (for example, moving from binary relevance to classifier chains/powersets etc) For now, just hoping to limit the decision to binary relevance machine learning solution vs just filtering the data. I'll be implementing in a very similar way to how it is done here.

Answer 1

Overall I think your reasoning makes sense. Between the two options I agree that the multi-label setting is more appropriate, because it doesn't artificially force a single label by document.

While this is not exactly your question I'd like to suggest a few other ideas. Keep in mind that they are not necessarily better, just potentially interesting to consider as well.

Topic modelling could be an option. Advantages:

• It's unsupervised: it just "follows the text", so there's no risk of any bias introduced in the training data or labeling method.
• LDA is the standard method, it's parameterized which means that you choose the number of clusters/topics. You could generate 2 topics and manually check the top words associated with each topic in order to know which one corresponds to which label.
• It's probabilistic, so for every document you would obtain a probability that the document belongs to topic A or topic B. This could be considered equivalent to multi-label, in the sense that a probability around 50% means that the document has the two labels.

However there's a risk that the topics would simply not correspond to your expected labels, it depends on the data. In this case it might be interesting to split into more than 2 topics, and then analyze the resulting topics and possibly label them.

Other idea: the design that you propose made me think of semi-supervised learning: given a small initial set of labeled data (initial training set) and a large set of unlabelled data, train and predict iteratively and add the newly labeled instances for which the model is most confident to the training set. There is also active learning, which aims to minimize the number of instances which need to be manually annotated. I'm not sure if these ideas are suitable for your case, mentioning just in case.

Finally in order to study the importance of words with respect to topics or documents TFIDF is too coarse in my opinion. Instead you could consider:

• Conditional probabilities: for instance $p(w|t)$ is the probability that a document of topic $t$ contains the word $w$, while $p(t|w)$ is the probability among documents which contain $w$ of topic $t$. Advantages: simple to calculate, intuitive and informative.
• For measuring the association between for instance a word $w$ and a topic $t$ (or two words), pointwise mutual information is quite standard.