I can't discuss my actual dataset, so please bear with me.
Let's say I have a dataset that contains a population of 20,000 examinations by a school principal. The principal is to record their examinations of student misconduct incidents. I want to implement NLP that assess the quality very broadly into two categories: "good examination" or "bad examination" of the full population.
An example of "bad examinations" are:"examination results - negative" or "exam results: negative". Or "check student's bags, checked the person. Nothing suspicious found. Or examination results negative". Or "Examination results positive". Or "ABC examined, results negative". ABC could be an abbreviation of the person's name.
A good examination would be where there is a lot of context: "Checked the student's bag and found textbooks, pencils, erasers, binders. No hidden compartments found. Interviewed the student and asked "x", "y", "z" questions. Her story corroborated other reports. Student presented herself in a clam manner. Examination results negative". Other times it could be paragraphs and paragraphs, and at the end "examination negative" or "examination positive"
There are also instances where all what could be listed is "wrong person because of different birth date. Examination results negative" and this is perfectly fine. Would this be a third category?
How would I go about implementing a reliable NLP solution? My first instinct is to take a random sample, classify it manually, and then apply it to the rest of the 20,000 records?
