I’m working on a social science research project that involves a Natural Language Processing methodology. I’m assigning multiple labels (For example, label 1:
Blockchain, Label 2:
Democracy) to news articles that describe their content. I’m then running them through a couple of Python sentiment packages and I’ll be analysing changes in the correlation between different labels over time and what their sentiment looks like.
For example, if I retrieve articles from 2006-2016, i’d expect to see nothing about Blockchain until 2008, then I’d expect to see a growing number of blockchain related articles. After the launch of the Ethereum Blockchain in 2014, I’d expect to see a growing correlation between blockchain labels and democratisation labels because the Ethereum virtual machine facilitates “decentralized finance”, accessibility to markets, technology etc etc.
One of my learning objectives is to bolster my Python knowledge but I need to justify any research design decisions. I don’t want an unnecessarily complicated methodology for the sake of using elaborate Python coding and fancy sounding libraries.
For this project, is the use of supervised machine learning meaningful/justified/necessary or should I keep things simple? Please can you provide thoughts on the below considerations? Anything else I should bring into this assessment?
Simple Approach to Multilabel Classification
- Two labels: 1. Blockchain 2. Democracy.
- Create lists of terms associated with each of these labels. For example, the “Democracy” label might include: Democra (which would pull democracy, democrat, democratisation etc.), vote, voting, liberal, presidency, parliament, representative government, etc.
- Filter my entire list of 5000 news articles for text that contains these terms, tag the articles if they contain any single instance of any of the terms. Tag each article with multiple labels if they contain terms relating to either label.
- Manually audit the results by randomly selecting 100 articles, reading them in full and judging the labels. This may result in further manual correction, for example, if the term “liberal” returns numerous articles when the term is not used in the context of democracy, I might then decide to manually check the context of all instances for that term..
- Pros and Cons: it will be impossible to manually review my filters in full. The existence of one instance of a term (an article mentions the word “democracy” once) doesn’t necessarily mean an article is about that subject (the threshold for receiving a particular label is very low). All terms will be considered unlike with the machine learning process where the training set might not include every term. The filtering/labelling of articles could be automated and is therefore scalable.
Advanced Approach to Multilabel Classification
- Two labels: 1. Blockchain 2. Democracy
- Create a dataset for training and testing a multilabel classification prediction algorithm: Manually tag a subset of 500 news articles (10% of the total population) with the two labels. This would be done using the simple approach described above combined with a manual audit of labels that this produces. For example, if the training/testing subset consists of 500 articles, I’ll read 100 of them to explore nuances and correct the labels. This may result in the decision to manually audit all 500.
- Run a TFIDF function to assess the importance of all words in each article
- Use binary relevance to assess each label independently with a Naive Bayes Algorithm for the classification.
- If the testing yields decent accuracy results, then use the model for the remaining 4500 articles
- Pros and Cons: this process will assess the importance of all words in each article and more broadly consider phrases that include my search terms. The labelling of this test/training data will be of higher quality than the labelling of all 5000 articles in the simple approach (manual review time focussed on 500 articles rather than 5000). If the training data doesn’t include particular terms however, then they will be dropped from the prediction process. Once the model is adequately trained, I can scale the size of this research without incurring additional work. For example, I could run 10,000 articles through the prediction algorithms.
My preliminary answer and chosen approach
Multilabel classification prediction algorithm (machine learning) not necessary but advantageous for the following reasons:
- the importance of all words will be assessed
- The labelling of training data can be thoroughly reviewed manually
- The main downside is that the training data might not include all terms considered relevant to a particular topic but I can address this by inserting articles about those topics or by fabricating them.
- I appreciate there are levels to the multilabel classification problem (for example, moving from binary relevance to classifier chains/powersets etc) For now, just hoping to limit the decision to binary relevance machine learning solution vs just filtering the data. I'll be implementing in a very similar way to how it is done here.