Best approach for rule-based system in multilabel classification-problem?

Question

I’m new to the world of NLP and am looking for some guidance. I want to create a rule-based system that “grades” text in accordance to some set of criteria. For example, one criteria could be “The author mentions that he/she wants money”, another “The author mentions working toward promotion”.

My initial idea was to use some available, open-source NLP-model, such as en_core_web_lg from the spaCy library. With such a model I could look at all verbs in a text, and classify texts as adhering to certain criteria when they have an appropriate verb with appropriate subject and object. I’ve read somewhere that exploiting the linguistic structure of sentences is a bad/unreliable way to go about things. The problem is that I don’t have any substantive data so as to allow supervised learning.

How do one typically go about creating a rule-based system for such a task? Is there any name for the problem I want to solve, maybe “Multi-label classification”? Any resources you could point me to?

Help a noob out!

I greatly appreciate it.

Answer 1

I think the first step should be to define the task more formally: do you mean there should be a grade for every such criterion? Is the set of the criteria fixed or an input parameter?

From the point of view of people understanding what you want to do you should also mention how long is a text, and how many texts, how many criteria? And if possible add an actual example with its expected output.

The problem is that I don’t have any substantive data so as to allow supervised learning.

This is a serious issue not only because you can't do supervised learning, but also because you can't evaluate the system. Evaluation is a must: if you don't know how well your system works, you can't guarantee anything about its output... so it's pointless to use it.

You should probably manually annotate a sample yourself. It might feel boring but it's actually useful indirectly for you to design the task correctly, because it forces the annotator to think about the details.

I’ve read somewhere that exploiting the linguistic structure of sentences is a bad/unreliable way to go about things.

I don't know where you read that but this is wrong. This approach might not be optimal compared to modern ML methods, but it can be a perfectly decent method.

Is there any name for the problem I want to solve, maybe “Multi-label classification”?

“Multi-label classification” represents a broad type of tasks, and that's not even the correct type if you want to predict grades :) Grades are numerical values so yours would be a regression task, as opposed to classification.

Anyway this is not the name of a specific problem, and it's very unlikely that your problem has a standard name or method.

How do one typically go about creating a rule-based system for such a task?

That's the design part and it's not easy. You need to study examples, try to find the clues that a human would use to decide, then try to transform these clues into actionable rules.

To be honest, a rule-based system for some kind of highly semantic and interpretative task is unlikely to perform very well, but why not try.