BRAT (brat rapid annotation tool) can be used for named-entity annotation:
Can BRAT be used for text classification annotation? I.e., given the text, annotate whether it belongs to some classes?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
BRAT was not designed with classification tasks in mind.
Like Valentin mentions, Prodigy is a more modern annotation management systems that fits well in a classification task workflow. We've been testing it at work - It is promising, but also has its limits.
You mention the potential need for crowdsourcing - I'm sure you know about AWS Mechanical Turk. You also talk about data/annotation integrity validation - Your best bet might very well be to develop your own annotation management system to cater to these needs. We have very similar requirements, and this is the route we've chosen to take.
We evaluated Prodigy before its 1.0 release - It had quite a few quirks, plus not all features were available at the time.
It seems to be very good at quick prototyping and building MVPs, but less adapted at serious large-scale annotation tasks. It's great to be able to answer quickly YES/NO to machine-generated annotations, but in our experience in many tasks those are not accurate often enough and it would be more efficient to manually annotate. It seems we can now load our own models into it, so there would be a possibility to improve that if we would invest dev time into Prodigy's framework.
Another very important aspect missing is any kind of distributed annotator management system - Multi-users, conflicts resolutions, etc. Add to that no support for integrity/validation checks. Prodigy team says it's all in the works though:
We're also working on an extension library, the Prodigy Annotation Manager, which will integrate with Prodigy and will let you set up complex annotation projects, manage multiple annotators, enforce quality control and keep track of the progress via an admin console. You can sign up to our mailing list to be notified about the private beta.
Edit: Just stumbled on WebAnno, which might be worth a look.