3
$\begingroup$

BRAT (brat rapid annotation tool) can be used for named-entity annotation:

enter image description here

Can BRAT be used for text classification annotation? I.e., given the text, annotate whether it belongs to some classes?

Improve this question
$\endgroup$
4
$\begingroup$

According to their documentation , BRAT does a lot of things, but text classification isn't one of them. BRAT is "too powerful" for that. I'd recommend you use a tool like Prodigy instead.

This should do what you're looking for.

Improve this answer
$\endgroup$
2
$\begingroup$

BRAT was not designed with classification tasks in mind.

Like Valentin mentions, Prodigy is a more modern annotation management systems that fits well in a classification task workflow. We've been testing it at work - It is promising, but also has its limits.

You mention the potential need for crowdsourcing - I'm sure you know about AWS Mechanical Turk. You also talk about data/annotation integrity validation - Your best bet might very well be to develop your own annotation management system to cater to these needs. We have very similar requirements, and this is the route we've chosen to take.

We evaluated Prodigy before its 1.0 release - It had quite a few quirks, plus not all features were available at the time.

It seems to be very good at quick prototyping and building MVPs, but less adapted at serious large-scale annotation tasks. It's great to be able to answer quickly YES/NO to machine-generated annotations, but in our experience in many tasks those are not accurate often enough and it would be more efficient to manually annotate. It seems we can now load our own models into it, so there would be a possibility to improve that if we would invest dev time into Prodigy's framework.

Another very important aspect missing is any kind of distributed annotator management system - Multi-users, conflicts resolutions, etc. Add to that no support for integrity/validation checks. Prodigy team says it's all in the works though:

We're also working on an extension library, the Prodigy Annotation Manager, which will integrate with Prodigy and will let you set up complex annotation projects, manage multiple annotators, enforce quality control and keep track of the progress via an admin console. You can sign up to our mailing list to be notified about the private beta.

Edit: Just stumbled on WebAnno, which might be worth a look.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ Added more details in my original answer since comments are limited to 600 chars. $\endgroup$ – Matt Fortier Jan 5 '18 at 14:53
  • 1
    $\begingroup$ If you ever come across a better tool, please let us know here! $\endgroup$ – Matt Fortier Jan 8 '18 at 21:00
  • $\begingroup$ Will do.~~~~~~~~~~~~ $\endgroup$ – Franck Dernoncourt Jan 8 '18 at 21:03
1
$\begingroup$

I think brat that fit some tasks as NER, PosTag...

with classification problems, we can add keyword to text which separate with tab symbol. After, we can annotate for label. for example: label sentence we can annotate for word: label, with some classes Thanks

Improve this answer
$\endgroup$
  • $\begingroup$ thanks. I was looking for a tool that has a bit more features such as making crowdsourcing easy, or ensuring the worker enters a valid label. $\endgroup$ – Franck Dernoncourt Apr 21 '17 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.